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ABSTRACT 

The issues, options, and procedures for annually 
measuring overall payment error in the Pell Grant program are 
specified in detail. Guidelines for establishing a definition of Pell 
Grant payment error are provided, and the design issues related to 
error measurement are examined. A comparison is made of options for 
selecting a study sample and for collecting data required to measure 
payment error. Data collection procedures are specified in detail, 
along with procedures needed to create a study database from the 
collected data. Guidelines for data analysis are also included. The 
correct award to students is based on such factors as enrollment 
status and student costs. Error measurement in the Pell program 
involves time-related research design factors, all of which must be 
understood when planning the data collection effort and when 
analyzing the data. Data collection activities are outlined for 
sample selection, student/parent interviewing, collecting hardcopy 
secondary data, and collecting data from institutions. Tasks and 
procedures required to create the database include: receipt of data, 
editing and coding, data entry, machine edit and updating, 
reformatting files for analysis package, producing marginal 
tabulations, merging data files, and reviewing case for quality 
control. Appended are; a glossary; a description of measurement 
tolerances of error in the program; algebraic specifications of error 
measures; and an estimate of requirements for conducting an 
assessment of overall payment error in the Pell Grant program. 
(SW) 
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EXECUTIVE SUMMARY 

Measuring total payment error in a social services 
program — particularly in the Pell Grant program— is a complex 
undertaking requiring a thorough understanding of the program 
itself and the various methodological issues involved* In this 
report r the issues / options, and procedures for axxnualXy measur- 
ing^ overall payment error in the Pell Grant program are speci-* 
fied in detail* The specifications developed here draw upon the 
experience gained in conducting a nationwide error study during 
Stage One of the current Pell Grant Quality Control project. 

The organization of this report follows the logical sequence 

of a quality control study. After a brief introduction in 

Chapter 1, each chapter is devoted to a major study phase: 

e Chapter 2 presents guidelines for establishing a defi- 
nition of Pell Grant payment error and examines the 
critical design issues related to error measurement. 

• Chapter 3 presents and compares options for selecting a 
study sample axvS collecting the data required to 
measure payment error* 

• Chapter 4 specifies in detail the procedures required 
to collect the data* 

# Chapter 5 specifies in detail the procedures required 
to create a study data base from the collected data. 

# Chapter '6 provides guidelines for analyzing the 
collected and prepared data# 

The following sunjmarizes ^ by chapter/ the general features 
of the specifications. 

Chapter 2 — Guidelines for Developing Measures of Er ror 

e Overall payment error in the Pell program is defined &s 
the difference between the zurount which should have 
been awarded and the amount Mftiich was actually 
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disbursed* The correct amount is based on the true 
values for the application items, enrollment status, 
cost of attendance, and the other factors which deter- 
mine a student's eligibility for a grant. 

Specification of operational definitions of payment 
error involves several serious methodological issues. 
It is important that analysts and policymakers under- 
stand each of these issues thoroughly when interprc»ting 
survey findings. Each of these issues revolves aro and 
the difference between "measured" error and "true" 
error. The issues include: 

The distinction between investigatory and confirm- 
atory measures of error 

The distinction between errors of omission auid 
errors of commission 

The distinction between verifiable and nonvarifi- 
able evidence 

In order to identify corrective actions, it is desir- 
able to decompose total payment error and allocate the 
resulting components to specific data elements and 
actors in the Pell Grant delivery system. Total pay- 
ment error can be divided into two •najor components; 
error attributable to the student and error attribut- 
able to the institution. Both student and institu- 
tional error can be further divided into policy- 
relevant subcomponents, such as error attributable to 
student misreporting of income. 

Error measurement in the Pell program involves the 
following three types of time-related research design 
factors, all of which must be understood when planning 
the data collection effort and when analyzing the 
survey data: 

The possible, impact of the timing of data collec- 
tion on the ability of students or parents to pro- 
vide accurate documentation of application data 

The impact of institutional validation and account 
reconciliation on grant accounts 

The impact of Pell program procedures, from appli- 
cation processing edits to account reconciliation, 
on grant levels 

At a minimum, the following information must be col- 
lected on each sampled recipient in order to measure 
total payment error: (1) the actual Pell disbursement. 
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(2) th9 Student Md Index (SAX) used to calculate the 
disbursement^ (3) the cost of attendance, (4) the 
enrollment status (5) documentation that verifies the 
. student's categorical eligibility for a Pell Grant, and 
(6) documentation that verifies application data. 
Items one through five must be collected from the 
institution and item six from the student and his or 
her parents and/or from agencies and organizations such 
as the IRS. If a "post-reconciliation" measure of 
error is desired, certain information may be collected 
from Program Information and Monitoring System (PIMS) 
records and from the confuted applicant record main- 
tained by the central application processor. 

e There eure two challenges to the validity of sample sur- 
veys; experimental bias and nonresponse bias. Ana- 
lysts and policymakers must be aware of any bias When 
interpreting study findings. Assessing the existence 
of experimental bias requires collection of data from 
nonsaunpled students at selected institutions and from 
students of nonselected institutions . 

Chapter 3 — Data Collection and Sampling Alternatives 

e Pour options can be identified for collecting the data 
required to meesure overall payment error: 

Option 1 : student record data would be "rollected 
by site visits to institutions, students and 
parents would be interviewed in person, and docu- 
mentation would be collected by n^il from the IRS 
and other agencies. 

Option 2 t would have the same features as Option 1 
except that in-person interviews with students and 
parents would be replaced by telephone interviews. 

Option 3 1 wuld be a scaled-down version of 
Options 1 and 2 — no student and parent interviews 
would be conducted. 

Option 4 : would have no field work; all institu- 
tional and student/parent data %»rould be collected 
by mail. 

e Although all the options have points in their favor, 
Option 1 is the most desirable approach since, of the 
four, it would provide the most reliable error 
measures . 
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• Two options can be identified for selecting a nation- 
ally representative sample of recipients. 

Option 1 : secure a list of each institution's 
recipients and sample from each list at the study's 
main office* 

Option 2 1 train field staff to draw a saraple at 
each institution* 

• The number of students who must be sampled depends on 
four factors I the degree of precision wanted for the 
statistical inferences derived from the data; the 
amount of confidence desired in these estimates 7 the 
degree of clustering to be used; and the degree of 
homogeneity within each cluster* The greater the pre- 
cision desired, the larger the sample must be; liXe- 
wise, the higher the confidence level desired, the 
larger the sample must be. Larger cluster sizes (i.e., 
the number of students at the saune institution) make 
the institutional data collection more efficient, but 
there is a loss in accuracy. Higher levels of homo- 
geneity among the students in each institution require 
larger samples for the same accuracy. 

Chapter 4 — Data Collection Specifications 

• The tasks, subtasks, and procedures required for com- 
pleting data collection Option 1 (site visits to insti- 
tutions, in-person student/parent interviews, c ^ col- 
lection of verifying documentation by nsail) and sampl- 
ing Option 1 (select sample at project office) follow: 

Ta sk 1; Select Sample 

Task 1.1: Select Institution Sample 

1. Detarrai.ne the Sampling Frame 

2. Stratify the Sampling Frame 

3. Cluster the Sampling Frame 

4. Determine the Size of the Sample 

5. Allocate the Sample among Strata and 
Clusters 

6. Draw the Sample 

Task 1.2: Select Student Sample 

1. Determine the Size of the Sample 

2. Determine the Sampling Frame 

3. Compile the Sampling Frame 

4. Stratify the Sampling Frame 

5. Draw the Sample 
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Interview Student.s and Parents 



Task 2.1: Develop Data Collection Instruments 

1. Specify Measurements and Measurement 
Levels 

2. List Data {Seeds 

3. Collate Stage One Instruments 

4. Evaluate the Stage One Instruments 

5. Write Revised Draft Instriaments 

6. Test Instruments 

7. Revise Instruments 

8. Secure FEDAC Approval 

9. Produce Final Instruments 

Task 2.2: Recruit Interviewers 

1. Determine Intezrviewer Qualifications 

2. Determine Number of Interviewers Needed 

3. Determine Distribution of Interviewers 
4 Recruit Applicants 

5. Screen and Interview Applicants 

Task 2.3: Train Interviewers 

1. Develop Training Manuals 

2. Schedule Training and Notify 
Interviewers 

3. Send Training Material to Interviewers 

4. Train Field Supervisors 

5. Train InteiTviewers 

Task 2.4: Contact Students and Parents 

1. Draft Letters and Forms 

2. Ccmpile Mailing List 

3. Send Letters and Forms 

Task 2.5i Conduct Interviews 

1. Assign Students and Parents to 
Interviewers 

2. Schedule Interviewers 

3. Deal with Refusals and Avoiders 

4. Conduct Intervievrs ^ 

5. £dit Questionnaires 

6. Return Questionnaires 

Task 2, 6: Implement Quality Control and 
Supervision Plan 

1. Coordinate Interviewers 

2. Edit Questionnaires 

3. Verify Interviews 

4. Supervise Interviewers 

5. Assign New Cases 

6. Control Expenses 
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TasX 2.7: Follow up by Talephone 

1. Send Instruments to Telephone Station 

2. Telephone Respondent or Interviewer 

3. Code Response 

TasX 3; Collect Secondary Data 

Task 3.1 I Collect Hardcopy Secondary Data 

1. Request SARs from institutions 

2. Telephone Institutions That Do Not 
Respond 

3. Review and Key Enter SAR Data 

4. Send Release Forms and Other Information 
to Students and Parents 

5. Telephone Students and Parents That Do 
Not Respond 

6. Obtain IRS Forms 

7. Obtain Tax Assessor Records 

8. Obtain Documentation from Financial 
Institutions 

9. Obtain Documentation from Public 
Assistance Offices 

Task 3.2: Conduct IRS Tape Match 

1. Create Tape to Submit to IRS 

2. Submit Tape and Release Forms to IRS 

3. Merge Returned IRS Tape with Study Data 

Task 4: Collect Data from Institutions 

Task 4.1: Develop Data Collection Instruments 

1. Design First Draft of Instruments 

2. Develop Other Data Collection Materials 

3. Contact Institutions for Field Test 

4. Conduct Field Test 

5. Revise Data Collection Instruments 

6. Conduct Second Field Test 

7. Revise Data Collection Instruments 

8. Prepare and Submit Clearance Package 

Task 4.2: Schedule Site Visits 

1. Send Initial Contact Letter to 
Institutions 

2. Send Second Letter to Financial Aid 
Officers 

3. Racontact Institutions that Fail to 
Submit Recipient List 

4. Construct Initial Master Site Visit 
Schedule 

5. Telephone Institutions to Arrange Visits 

6. Adjust Site Visit Schedule As Necessary 
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Task 4,3: Recruit Field Representatives 
!• Advertise for Financial Aid 
Professionals 

2. Interview by Phone 

3. Interview in Person 

Task 4,4: Train Field Representatives 
1* Develop Training Manuals 
2. Contact Institutions to Arrange Field 
Practice 

3* Mail Training Manual to Field 

Representatives 
4« Conduct Training Session 

Task 4- 5 J Conduct Site Visits 

1. Prepare for the Site Visit 

2. Meet with Financial Aid Director 
3» Complete File Abstracts 

4» CcMnplete Control Group Forms 

5, Conduct Exit Interview 

6, Edit Completed Instruments 

7. Complete Transmittal Form 

8. Bundle Data and Insert in Pre-Addressed 
Mailer 

9« Seal/ Tape, and Mail Data Package 
10 « Record Mailing Data and Location in 

NotebooX 
11* Call Supervisors 

Task 4.6: Implement Quality Control and 
Supervision Plan 

1. Send Periodic Memoranda 

2« Establish Telephone Schedules 

3. Validate the Field Representative's Work 

4. Observe Field Representatives On-Site 

Task 4.7: Follow up by Telephone 

1« Telephone Field Representative or 
Institution 

2. Code Instrument 

Task 4.8{ Debrief Field Representatives 

1. Plan One--day Debriefing Session 
2* Conduct Debriefing Session 



Chapter 5 — Data Preparation Specifications 

# The tasks and procedures required to create the study 
data base follow; 

Task 1: Receive Data 

1» Record on Master Control Log 

2. Log onto Respondent Control File 
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3. Scan Edit 

4 . Batch 

5. Record on Master Control Log and Batch 
Control Log 



Task 2t Edit and Code 

n Code 

2 . Edit 

3. Conduct Quality Control Check 

Task 3: Key Enter 

1. Key Enter 

2. Key Verify 

Task 4t Machine Edit and Update 

n Run Edit Prograuti on Keyed Data 

2. Resolve Edit Failures 

3. Update File 

4. Repeat Edit Process Until File Is 
"Clean" 

5. Run Edit program to Ensure Completeness 
of Data 

Tadk 5t Reformat Files for Analysis Package 
n Create SAS File 
2. Assign Variable Labels 

Task 6: Produce Marginal Tabulations 
n Produce Statistics 
2. Review 

Task 7: Merge Data Files 

1. Code Dependency and Marital Status 

2. Prioritize Data 

3. Merge Files 

4. Compute SAI and Award 

Task 8t Conduct Quality Control Case Revie%/ 
n Select Sample 
2. Review and Update 

Chapter 6 — Guidelines for Conducting Data Analysis 

• Rules for including cases in summary statistical runs 
should be specified. These rules should address the 
following issues: 

Inclusion or deletion of cases where no Pell dis- 
bursements are recorded 

Inclusion or deletion of cases where data collec- 
tion is incomplete 

Inclusion or deletion of cases where verification 
data is incomplete 
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The most important descriptive statistics to be pro- 
duced are the univariate statistics whicli show the 
incidence and general form of overall payment error • 
Standard statistical software packages are able to 
generate frequency distributions and other univariate 
statistics and graphic representations of univariate 
distributions . 

The first step in investigating the causes of payment 
error is to determine whether specific student, insti- 
tution^ or program characteristics are associated with 
the incidence or sis&e of payment errors* Bivariate 
analyses — t%ro-way tables and statistics measuring the 
strength of association between two variables — can be 
extremely useful in both answering questions about 
sx>ecific relationships and in exploratory data 
analysis. 

Multivariate analysis is designed lo address more com- 
plicated questions of relationships between error and ^ 
characteristics of students, families, and institutions 
than is bivariatb analysis. Three purposes can be 
served by multivariate analysis: 

Testing of a priori hypotheses 

- Exploratory data analysis 

- Error-prone modeling 

Hypothesis testing is a methodological approach where 
hypotheses and theories are subjected to real world 
data in order to confirm or reject these hypotheses. 
Exploratory analysis, on the other hand, uses the real 
world data to develop the hypotheses and theories. 
Error-prone n»deling is a form of exploratory analysis? 
however, it differs from the other two in that its pur- 
pose is not to uncover relationships among variables 
but rather to split a sample into groups ^ere the 
observations in a group have as similar error levels as 
possible while error levels across groups are as dis- 
similar as possible. 
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CHAPTER 1 
INTRODUCTION 



BACKGROUND 

The Pell Gran'i program is the second largest student assis- 
tance program administered by the Office of Student Financial 
Assistance (OSFA) . In the 1981-62 academic year over $2 billion 
were distributed to roughly 2.7 million students attending eli- 
gible postsecondary institutions. A program of this size carries 
with it significant management responsibilities. One of OSFA's 
major goals is to ensure that the Pell Grant program operates in 
an efficient manner/ and that available resources are allocated 
properly to those students entitled to aid. Senior policymakers 
in Congress, the Office of Management and Budget (OMB), and the 
White House share the concern that Pell Grant funds be distri- 
buted in accordance with regulations and legislation. 

Quality Control (QC) — detecting and reducing error — is 
essential for the Pell Grant program. One purpose of Stage Two 
of the current Pell Grant Quality Control project is to design a 
QC system to continuously measure and analyze Pell Grant program 
performance. As one component of this overall QC system^, it 
is important that ED policymakers, 0MB, and Congress be provided 
on a routine basis {at least annually) with a measure of overall 
payment error in the program as input to policy and legislative 
decisions • 



^ Preliminary Quality Control System Design for the Pell 
Grant Program (Advanced Technology, 1982) provides a general 
design for all components in the Pell Grant QC system. 
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A QC study in 1978-79 and the 1980-81 study in Stage One of 
the current QC project provided such a comprehensive measure of 
Pell Grant payn^nt error. The studies # however/ were in essence 
''one-'time-^nly " contracted efforts. Currently ^ OSFA has no on - 
ijoinq system for collecting and analyzing information on overall 
payment error. While OS^^A's Division of Certification and Pro- 
gram Review (OCPR) performs a review function for all student aid 
programs # it gathers neither the type nor the quantity of infor- 
mation needed to accurately estimate total error in the program. 
In addition # OSFA receives and analyzes data from the Pell Grant 
application processor on application corrections made in response 
to system edits and to validation of application infonnation by 
postsecondary institutions • These data, however # provide only a 
very rough notion of the scope of applicant raisreporting and pro- 
vide no measure of institutional error. 

PURPOSE AND ORGANIZATION OF REPORT 

Obtaining an accurate # overall measure of payment error in 
the Pell Grant delivery system is a complex undertaking. The 
purpose of this report is to define the issues # options, and pro- 
cedures for annually attaining such a measure. The organization 
of this report follows the logical sequence of a QC study. In 
Chapter 2, guidelines for establishing a definition of payment 
error are presented. Various methodological issues are discussed 
including the distinction between confirmatory and Investigatory 
measurements, the measurement of errors of commission and omis- 
sion/ and the relationship between time of data collection in the 
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Pali Grant delivery cycle and measurement of error. In Chapter 
3/ four poesible approaches for collecting the required data are 
analyzed. One of the four is recoiranended for an ongoing measure- 
ment systexn—a three-faceted approach which includes: (1) visits 
to postsecondary institutions; (2) in-person interviews of grant 
recipients and their parents; and (3) collection of documentation 
from various governmental agencies, tax assessors / and banXs to 
veri^ student application information. Also in Chapter 3/ 
alternatives for selecting a recipient sample are examined, and 
the relationship of recipient sample size to the precision of 
estimates about error and the amount of confidence one can have 
in those estimates is explored* In Chapter 4, specifications for 
collecting data using the approach recommended in Chapter 3 are 
set forth in detail. This chapter is essentially a manual for 
conducting data collection. Required tasks have been identified- 
For each task are first, a discussion of the rationale for the 
task and any important issues that need consideration in imple- 
menting the task and^ second, a step-by -step list of required 
procedures. Chapter 5 provides det /^iled specifications for cre- 
ating the study data base. As in Chapter 4, tasks and procedures 
are identified. Chapter 6 provides guidelines for analyzing the 
collected and prepared Pell Grant QC data. Tne appropriateness 
and usefulness of various analytical techniques are considered* 
In addit^ion, a glossary in Appendix A defines terms as they are 
used in this report. The reader ma/ find it useful to scan the 
glossary before continuing further. 
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CHAPTER 2 

GUIDELINES FOR DEVELOPING MEASURES OF ERROR 



This chapter presents guidelines for establishing a defini- 
tion of overall payment error in the Pell Grant program and 
exeunines study design issues related to error measurement. This 
chapter focuses on six critical areas t 

• Basic Construct for Error Meas'jrement (What is a work- 
able definition of Pell Grant payment error?) 

• Issues in Establishing Operational Measures of Error 
(What are the critical design issues that must be 
considered?) 

• Decomposition of Overall Payment Error (How can total 
error be divided into policy-relevant subparts?) 

• Error Measurements over the Course of the Pell Cycle 
(What are the time-related research design issues?) 

• Data Sources for Pell Error Measurement (What data are 
required for measurement, and what is their source?) 

• Auxiliary Data Collection Requirements (What data are 
needed to measure experimental bias?) 

BASIC CONSTRUCT FOR ERROR MEASUREMENT 
Overall Payment Error 

Overall payment error in the Pell Grant program is generally 
defined as the difference between the correct awards (the amount 
which should h^ve been disbursed) and the awards Which actually 
were disbursed to students over the course of a program year. 
The correct value is based on the true values for the application 
items, enrollment status, cost of attendance, and the other 
factors Which determine a student's eligibility for a Pell Grant. 
Net overall payment error is the difference between all over- 
awards to recipients and all underawards, while absolute overall 



payment error is the stun of overpayments plus underpayments. As 
discussed below, any operational definition of overall payment 
error can only approximate "true" payment error in an entitlement 
program such as the Pell Grant program. It is not feasible to 
measure precisely the number or size of awards which could have 
been disbursed to eligible students but were not, due to error in 
ascertaining eligibility. Statistical estimates of error in the 
Pell Grant program are normally limited to estimates of error for 
the recipient rather than the population of eligible applicants. 
Further, the precision of estimates of overall payment error is a 
function of the extensiveness and intensiveness of the investiga- 
tion of student eligibility and institutional, processor, or 
vendor procedures. More stringent examination of eligibility 
and/or procedures is likely to produce higher estimates of pay- 
ment error. In princitle* the more intensive the research into 
the circumstances affecting each award, the closer the measure- 
ment of error will be to "tru^" error. 
Error Measurenwnt and Quality Control 

"True error" may not be the most appropriate reference point 
for a quality control system. Rather, the best measure of over- 
all payment error for QC purposes should be geared to concepts of 
student eligibility and institutional procedures which are amen- 
able to corrective actions and can be measured accurately and 
effectively. The precise definition of overall payment error in 
the Pell program should be formulated so that it provides accu- 
rate and reliable measures of program error which can be moni- 
tored in a quality control system. The definition of overall 
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payment error should include each of the components of program 
error that OSFA wants to monitor and to control or correct. 
Measures of the components of overall payment error, as well as 
measures of total payment error, provide necessary data for 
evaluating program performance from a quality control perspec- 
tive. Systematic use of these data will allow OSFA to identify 
areas in need of corrective action, and, in conjunction with ' 
appropriate analysis procedures, help to establish probable 
causes of error. After corrective actions have been instituted, 
quality control measures will allow OSFA to measure the effec- 
tiveness of corrective actions as well as to identify any new 
problems arising in the Pell system. 

ISSUES IS ESTABLISHING OPERATIONAL MEASURES OF ERROR 

Specification of operational definitions of payment error to 
be used in a quality control system for the Pell Grant program 
involves several serious methodological issues . Each of these 
issues revolves euround the difference between "measured" error 
and "true" program error. Although it may not be possible to 
obtain the roost accurate, or closest approximation to "true** 
error, analysts and policymakers should understand each of these 
issues thoroughly in order tJ> properly analyze survey findings. 
Investigatory Versus Confirmatory Measures 

Perhaps the roost important methodological decision to be 
made in determining definitions of error in the Pell Grant pro- 
gram is the decision as to whether the verification of grant 
eligibility data is to be based on thorough, independent 




investigation of student and/or institution procedures , or 
whether verification is designed to obtain data from students (or 
institutions) Which substantiate data previously supplied by 
them. The first option, investigatory measurement, requires 
costly and time-consuming checks of student finances including 
collateral or credit checks, examination of tax files, cross 
checks of educational histoxry, and so forth. This type of inves- 
tigation would establish, with a high degree of certainty, stu- 
dents' "true" financial status, family circumstances, and educa- 
tional status related to Pell eligibility. Confirmatory measures 
are designed only to verify that application data provided by 
students or institutions can be substantiated. Thus, if a stu- 
dent reports an adjusted gross inc<»ne (AGI) of $10,000 in his or 
her application for a Pell Grant, a confirmatory error measure- 
ment procedure would involve verifying that this AGI was in fact 
reported on the appropriate student or family tax form. An 
investigatory procedure would determine whether this figure of 
$10,000 was the correct AGI figure. In short, confirmatory pro- 
cedures are designed only to substantiate or validate data pro- 
vided by applicants or institutions, while investigatory proce- 
dures are designed to independently establish the facts relating 
to grant eligibility. Investigatory measures of error are there- 
fore fundamentally different from confirmatory measures. 

The method of comparison between reported and verified 
data — investigatory or confirmatory — must be specified for each 
data Item being evaluated in any study of error in the Pell 
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program prior to the planning of data collection efforts • 
Measures of error in the Pell program in 1978-79 and in 1980-81 
studies were confirmatory measures* 
Errors of Omission Versus Errors of Conmission 

The issue of v^ether data submitted by applicants for Pell 
Grants represent a true picture of their financial status in- 
volves* the related c[uestions of accuracy and ccxnpleteness • 
Errors of onission by applicants are those where an applicant 
failed to report pertinent data* Errors of commission are those 
Where data were reported incorrectly. These errors are closely 
related to the issue of investigatory versus confirmatory 
measurement discussed above* 

Suppose, for example, that a student reports only part of 
net family assets. He or she correctly reports having $2,000 in 
a savings account but neglects to report $20,000 in a trust fund. 
This student has OTiitted pertinent data which might never be dis- 
covered in the course of a confirmatory measure of error # The 
student can confirm that the correct amount in his or her savings 
account is $2,000, but if the data collector does not )cnow that 
the student has a trust fund, the issue of confirming the size of 
the trust fund may never arise. 

In the course of conducting dar^ collection for a quality 
control study, interviewers should be extremely careful in phras-- 
ing questions so that the risk of omission of pertinent data is 
minimized. Analysts should remain aware of the fact that the 
verification of data submitted by students is substantively 




different from the establishment of the ''true'' data Which vAsuld 
include data inadvertently or intentionally omitted by students. 

Finally, errors of commission (instances Where students have 
submitted incorrect data) will :e caught only When verification 
data are available. In cases \^ere confirmatory measures are 
being used, verification data can be supplied, directly or 
indirectly, only by the student and his or her parents. If the 
Pell applicant or his or her parents claim not to have filed an 
income tax return for the prior year, it is extremely difficult 
to verify what the correct AGI actually is. It is far easier to 
establish a measure of error among Pell recipients who freely 
provide documentation than among those who provide little or 
none. Analysts should be aware of this fact, and alternative 
approaches to estimating error among the recipient population for 
whom documentation has not become available need to be given 
careful thought. 

Verifiable and Nonverifiable Data 

The Pell Grant application includes several questions which 
require students to project future income and/or family circum- 
stances. Perhaps the roost significant of these items is the num- 
ber of family members Who will be attending poatsecondary insti- 
tutions in the fall. Students filling out applications can make 
a reasonable guess as to this figure, but often circumstances 
change. Analysts may therefore wish to consider the verification 
of this item in a special light and not automatically consider 
discrepancies between the application figure and the actual num- 
ber in postsecondary education as errors. 
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other types of estimates, such as student inc<^e data or 
nontaxable income data for the coming year, are essentially non- 
verifiable, in that assessing the accuracy of these estimates is 
somewhat arbitrary. The figures can be verified the following 
year, but it is extremely difficult, if not impossible, to estab- 
lish the reasonableness of student-provided estimates. In any 
case, such "error" is not significant in terms of award eligi- 
bility or program error, since projected figures are not used in 
Student Aid Index (SAI) computations. In cases Where applicants 
are estimating prior year inccsoe, however, error can be signifi- 
cant. This happens in cases \^ere grant applications are sub- 
mitted before t£uc forms have been filed. In cases v^ere tax form 
data are estimated, discrepancies between application data and 
tax data submitted after the application has been filed may 
reflect circumstances the applicant did not anticipate, but Which 
nevertheless should have been reportCNl in corrections to the 
application. Analysts may therefore choose to differentiate 
error in projected data or estimated data from other errors, but 
there is a strong case to be made for including any discrepancies 
in estimated tax form data and subsequently filed tax form 
Qocumentation as application error. 

The most problematic type of estimated data is the data pro- 
vided by students filing special condition (or "supplemental") 
forms. These students submit that substantial changes in family 
and/or financial circumstances have made their prior year's 
incomes inaccurate representations of their current statuses. 
Students therefore provide estimates of current year income in 



their special condition applications. Such data cannot be veri- 
fied by norroal procedures. Further/ verification of the accuracy 
of students* estimates does not have the same meaning as verifi- 
cation of reports of prior year income. Matches between e^ti- 
xnates or projections and data on tax forms filled out subse- 
qv;ently probably should not require the same level of precision. 
Analysts should specify how special condition filers should be 
treated in error studies. If such recipients are not subject to 
verification it may be desirable to impute error for this sub- 
population in total error estimates for all recipients. 
Types and Strengths of Documentary Evidence 

Verification of student application data may be obtained 
from a series of sources — interviews with parents and students/ 
documentation obtained from student aid files at institutions/ 
copies of tax forms submitted to the IRS, documentation of home 
value obtained from tax assessors/ etc. In some cases more than 
one documented value for a given data element on the Pell appli- 
cation will be collected. The value of measured error or these 
items will then depend on which documented value is selected as 
the "verified" value. Analysts must therefore devise an algo- 
rithm to be integrated into the data analysis programs for 
selecting the best information to be used in verification of 

application data. 

The first step in determining the priority of verification 
data is to create a matrix listing all the application items down 
the vertical axis and all the possible sources of verification 
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across the horizontal axis. Types of verification can be subdi- 
vided according to the strength pr reliability of data collected 
in a single instrument or data source if this is appropriate. 
For instance, in interviews, parents cam provide copies of notar- 
ized forms, signed forms, unsigned wrlcsheeta , or simple verbal 
assertions as documentation of adjusted gross income. These dif- 
ferent levels of documentation may be listed as different cate- 
gories across the horizontal axis. Once all the possible sources 
of verification have been specified, the relative strength of 
each can be ranked for each data item. The rules for remking 
verification data should be logical and consistent but ro2»y vary 
for different types of data. 
General rules might be: 

1. Certified IRS tax form documentation overrides non- 
certified IRS data. 

2 Tax form documentation obtained directly from the IRS 
overrides tax form data from any other source. 

3 . Notarized documentation overrides nonnotarized docu- 
mentation (except for certified data obtained from the 
IRS) . 

4. Signed forms override nonsigned forms for documenting 
any data elemeftt* 

5. Parent data override student data on all dependency 
status questions. 

6. Parent data override student data on all family finance 
items for dependent students. 

7. Student data override parent data in questions relat- 
ing to student (or student and spouse) earnings and 
assets . 

8. Unsigned parent or student data override unsigned data 
found in student records. 

o 
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An example of a completed data source priority matrix (used 
in the Stage One study) is presented in Chapter 5. 

DECOMPf.SITIOU OF OVERALL PAYMENT ERROR 

For the current Pell Grant applicant and award calculation 
system the correct calculation of a Pell Grant depends on up to 
35 pieces of information or data elements, enumerated in Figure 
2-1. The first 24 items are used to calculate the SAI, Which is 
combined with the cost of attendance and enrollment status to 
calculate the expected award. The lasr. nine items are categori- 
cal criteria which must be satisfied, or any payment made is 
considered to be totally in error. If the application form were 
simplified, as was recommended in the Stage One QC Study then 
fewer pieces of information would be needed to make the correct 
calculation of a Pell award. 

In order to identify corrective actions, it is desirable to 
decompose total payment error and allocate the resulting com- 
ponents to specific data elements and actors in the Pell Grant 

delivery system. 

There are alternative levels of decomposition, as depicted 
in Figure 2-2. The first level decomposes overall error into two 
major components i student error and institutional error. 

Overall error is defined as the difference between the 
amount actually disbursed to a student and what would have been 



Ipor recommendations to improve management procedures in 
the Pell Grant program see Quality in the Basic Grant Delivery 
System, Volume 2, Corrective Actions . 
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1. Adjusted Gross Income (AGI) 

2. Taxes Paid 

3. ^4edical Expenses 

4. Father's/ Applicant's Income 

5. Mother *s /Spouse ' s Income 

6. Nontaxable Inc<Me 

7. Family Size 

8. Tuition 

9. HcHae Value 

10. Home Debt 

11. Other Assets 

12. * Other Debts 

13. Business or Farm Value 

14. Business or Farm Debt 

15. Nusober in Postsecondary Education 

16. Student's Net Assets 

17. Student's Net Income 

18. Student's Expected Income 

19. Support, Previous Year 

20. Support, Current Year 

21. Claimed as Tax Exemption, Previous Year 

22. Claimed as Tax Exemption, Current Year 

23. Lived with Parents, Previous Year 

24. Lived with Parents, Current Year 

25. Bachelor's Degree (BA) 

26. Citizenship (CIT) 

27. Statement of Educational Purpose (SEP) 

28. Financial Aid Transcript (FAT) 

29. Course Length 

30. Eligible Program of Study 

31. Grant or Loan Default 
3 2- Half-time Enrollment 

33. Satisfactory Academic Progress 

34. Cost of Attendance 
3 5. Enrollment Status 



FIGURE 2-1 
DATA ELEMENTS USED IN AWARD CALCULATION 
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FIGURE 2-2 
DECOMPOSITION OF TOTAL PAYMENT ERROR 
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disbursed if correct values were used for all 35 data elements *2 
Decomposition of this overall error depends on grouping the data 
elements by responsibility. The student is assumed to be respon- 
sible for the first 24 items (those used to calculate the SAI); 
the remaining 11 items are the responsibility of the 
institution. 

Student award error may exist %*honever 1 of the first 24 
data elements is in error. This error is allocated to the stu- 
dent because these data elements are provided to the Pell deliv- 
ery system by the student applicant. 

Student error is the difference between two calculated pay- 
ments. Both would be based on correct values for cost of attend- 
ance and enrollment status and the assumption that all categori- 
cal criteria are satisfied. However^ one calculated payment uses 
the value of the SAX on record at the institution, Vhile the 
other uses the value of the SAI based on the "best" data un- 
covered through the raultifaceted field work effort. The ration- 
ale here is that this difference is attributable to student error 
and misreporting as it affects the SAI.^ 

Institution error exists whenever cost of attendance and 
enrollment status are in error or When one of the nine categori- 
cal criteria is not satisfied. These errors are allocated to the 



^Algebraic formulations of overall payment error and its 
components are presented 2*s Appendix C. 

^Not all student application error affects SAI corapu*:a- 
tions? only error that affects SAI computations results in award 
error. 




institutions because they are responsible for either providing 
these data elements or in certifying that the categorical cri- 
teria are satisfied' 

Institution error is calculated as the difference between 
the amount disbursed by the institution and what would have been 
the calculated disbursement based on the correct values of cost 
of attendance, enrollment status, vrtiether the categorical cri- 
teria were satisfied, and the SAI on record at the institution. 
Here any differences in SAI are held constant, and what differ- 
ences exist are attributable to institutional mistakes in record- 
ing enrollment status and cost of attendance, or in certifying 
the satisfaction of the categorical criteria. 

Student error can, at the extreme, be broken down into 24 
more detailed components, one associated with each of the 24 
application data elements. These marginal components represent 
how payment error would be changed if only that data element were 
not corrected While all other data items were replaced with the 
correct or best values. It is also possible to define these mar- 
ginal components as the amount payment error would be changed if 
only that data element were corrected while all the other data 
items were not corrected. 

While decomposition at the first level creates two 
components — stadent error and institutional error — which when 
added up equal overall error, this additive property does not 
hold for the 24 marginal components of student error. This defi- 
ciency is attributable to the interdependency of the various com- 
ponents involved in the SAI calculation. 
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As indicated in Figure 2-2, it is also possible to define 
error attributable to logical and policy-relevant groupings of 
application items, such as those Which can be verified using an 
IRS form 1040, those detprraining dependency status, and those 
involved in the net worth ccmiponent of economic resources. 

There are three levels of decomposition of institutional 
error. First, institutional error is separated into two compon- 
ents: categorical eligibility error and calculation/ accounting 
error. Categorical eligiblity errors occur whenever field work 
and record checks show that one or more of the nine categorical 
criteria has been satisfied. At the extreme, categorical error 
can be decomposed into nine separate ccHoponents, one for each of 
the categorical criteria. As was true for student error, it is 
also possible to define error attributable to logical and/or 
policy-relevant groupings of these nine criteria. 

The second component of institutional error, calculation and 
accounting error, occurs Whenever there has been an error in 
correctly recording cost of attendance, enrollment status, and 
calculating and disbursing a correct award. As indicated in 
Figure 2-2, calculation and accounting error can be decomposed 
into three ccsnponentsj cost of attendance error, enrollment 
status error, and calculation error. These three components when 
added together are equal to calculation smd accounting error- 
However, categorical eligibility error and calculation and 
accounting error When added together do not equal institutional 
error . 



It ia very important whan analyzing the data to remember 
which con^nents possess the additive properties and which do 
not. Confusion on this point could lead to incorrect policy 
inferences and inappropriate corrective actions. 

An additional caveat concerning the error definitionp, dis- 
cussed above is that they are based on the full annual Pell 
cycle. In other words, they are defined as if the entire cycle— 
from application through account reconciliation — was complete. 
Annual error assessment is unlikely to be carried out over such a 
prolonged period. Therefore, error measurement and definition 
must take into account the differences in timing between the 
measurement of error and the Pell grant cycle, which is discussed 
in the next section. 

ERROR MEASUREMENTS OVER THE COURSE O F THE PELL CYCLE 

Error measurement in a Federal grant program such as the 
Pell prograir involves three types of time-related research design 
factors. The first is the possible impact of the timing of data 
collection on the ability of students or parents to provide both 
accurate application and corrections data and documentation of 
these data. The second is the impact of institutional validation 
and account reconciliation procedures on grant accounts. Fin- 
ally, Pell program procedures, from application processing edits 
through validation to account reconciliation, result in substan- 
tial adjustments to Pell final account figures. The first design 
factor, the timing of data collection on student error measures. 
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is discussed in greater detail in Chapter 4, but some basic 
inethodoXogical considerations are discussed below. Following 
this discussion, the importance of the combined impact of inter- 
nal procedures and ED procedures on error measures over the 
course of the Pell cycle is briefly reviewed. Specific measure- 
ment points for assessing error over the course of the Pell cycle 
are then presented. 

Time of Measurement and Student Error 

Students normally apply for Pell Grants in winter or spring 
preceding the school year for which they aeeik aid. As discussed 
above, some students or parents have not yet completed their tax 
forms at the time of application. Others may experience signifi- 
cant changes in their financial circumstan-es related to their 
grant eligibility between the time they file their applications 
and When they receive their first disbursements. Students may 
also voluntarily correct data submitted in applications because 
they are incorrect either because of applicant error or process- 
ing error. Finally, students may be required to correct appli- 
cation d»:*--a by institutions or by the Department of Education 
through the validation system. Depending on when information is 
collect<jd, therefore, different views of student error might be 
recorded because either (1) data to verify application data arc 
more readily available, ktiich affects error measures (as just 
described} or (2) the data submitted by the applicant have 
actually been changed. 
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Measures of application error « i.e., comparisons of initial 
application data and verified data, should be used to measure 
application error* These measures are necessary for monitoring 
quality of application forms ana forms processing. Measures of 
student error taken after the corrections process is completed 
should be used to assess the amount of error not removed by 
existing program procedures. 

Institutional and Program Procedures over the Course of the Pell 
cycle 

Pell disbursements are made at least twice during the course 
of an academic year and frequently more often. Institutions 
must follow program procedures for authorizing and disbursing 
grants, as well as their own internal program accounting and 
audit proceduces. Reconciliation of Pell accounts may take place 
at various points throughout the year. Final reconciliation data 
are normally submitted to the Department of Education following 
the final disbursements in spring or suitroer. The Program Infor- 
mation and Monitoring System (PIMS) runs institutional data on 
student accounts through a series of edits designed to identify 
errors in institutonal accounts reconciliation so that such 
errors can be corrected. Because there is an ongoing process of 
disbursements, accounts reconciliation, and review over the 
course of the year, measurement of institutional error is par- 
ticularly sensitive to the time factor in research design. On 
the one hand, it is extremely difficult to identify procedural 
err-)rs after the fact, i.e., after accounts have been reconciled. 
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This is particularly true for errors related to Pell Grant eligi- 
bility. Institutions are required to check students* enrollment 
status prior to disbursing grants. It is very difficult to go 
back at the end of the year and determine what a student's status 
at the time of disbursement was if the student drops or adds 
credits later in the academic term. It may also be difficult to 
determine if statements of academic purpose or financial aid 
transcripts have been collected if data collection is conducted 
late in the cycle, since these records may no longer be easily 
accessible, particularly in highly autcaaated systems. On the 
other hand > what may look like an over- or underpayment at one 
time may be straightened out by the end of the year. Therefore, 
as in the case of student error, different roeasur*»' of 
institutional error may be required for estimating overall 
payment error . 

Data Collection Points in the Pell Cycle 

Measurement of overall payment error in the Pell program can 
be taken at five basic points in the program cycle: 

• The point at Which the SAI is first calculated by the 
application processix^ contractor using application 
data and any necessary corrections. This value of the 
index is denoted as SAI (0) . 

• The point at ^ich the Student Aid Reports (SARs) is 
collected from the institution during the sampling 
stage of the study. The aid index recorded at this 
time is denoted as SAI ( 1 ) . Cost of attendance and 
enrollment status collected at that time are denoted as 
COST(l) and EKR0LI,(1), respectively. 

• The point when data collectors complete parent and 
student intervie%/s, collect releases for copies of tax 
returns, or perform whatever data collection is 
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appropriate for verification of eligibility data. 
These data are used to calculate a verified student aid 
index SAX (3). 

• The point ^en data collectors abstract information 
from student recorrds at the institutions. The aid 
index, coat of attendance i and enrollment status col- 
lected at this tiae are referred to as SAX (3), COST (3), 
and fiHROLLU)' Data should also be collected on actual 
and planned disburs«»nentS/ AD (3). 

• The point at Whidi institutions submit their final 
reconciliation rosters to the Department of Education. 
Values for the aid index / cost of attendance, enroll- 
ment status, and actual disbursements collected at this 
time are denoted as SAI(4), C0ST(4), ENROLL(4), and 

AD ( 4 ) , respectively . 

The specifications for develop^-ng measures of overall payment 
error from these data are described below. 



DATA SOURCES FOR PELL ERROR MEASUREMENT 

As discussed in the previous two sections, error measurement 
and definition depend on the timing of data collection said the 
intensiveness of data collection. Intensiveness of data collec- 
tion afflicts Which components of overall payment error can be 
assessed. In this section the data requirements for alternative 
error definitions are described. 

Decisions concerning error definitions for annual assessment 

of overall error would include: 

• Whether measurement should be before or after reconcil- 
iation, or both 

• Whether total error should be broken down into student 
and institutional components 

• Whether student error should be further broken down 
into item or marginal impacts and it«n error rates 

• Whether application item error rates should be 
measured 
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• Which items should be verified 

• Whether institutional error removed through reconcil- 
iation should be assessed 

Five sources of data can be defined: 

• Ccxnputed applicant record from the central proceesor 

• Verification of application information directly from 
the students and parents during interviews or indirect- 
ly from agencies su^ as the IRS 

• Institutional copies of SAR used for disbursement 

• Institutional record abstracts us«I to verify institu- 
tional data 

• Reconciliation roster data aiibmitted to ED 

Figure 2-3 indicates the relationships between data ele- 
ments and data sources. For exan^le^ information on an individ- 
ual recipient's cost of attendance is collected either at insti- 
tutions or from PINS's reconciliaticm roster. 

Figure 2-4, (tetrix of Outcome Measures and Data Sources, 
identifies the data collection sources required for the alterna- 
tive error measures. Entries in the table indicate the nature of 
the data required for the measure. For exan^le, estimation of 
pre-reconciliation, total award error requires student inter- 
views r institutional record abstracting of data on disbursements, 
cost of attendance, enrollment status, and categorical items. 

As indicated in the table, most measures require data from 
students and parents and institutional visits. However, if a 
restricted error definition, such as tax form error, was used, 
the student/parent data collection could be restricted to secur- 
ing tax form releases. Various data collection options are dis- 
cussed in more detail in Chapter 3. 
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Roster 



Income 

Expenses 

Wealth 

Status 

Family Size 
Structure 

Student Aid 
Index (SAI) 

Cost of 
Attendance 

Enrol Iment 
Status 

Categorical 
Rules 
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X 
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X 
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FIGURE 2-3 
DATA EI.EMENTS BY DATA SOURCES 
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OUTCOWE l«ASURES 



CENTRAL 
PROCESSOR 

Computed 
AppI icant ftecqrd 



First 



Tctal Award Error 
Pre^ocoocl I latloo 

Po8t-rocof)ci n at Ion 

Student Error 
Total Pre-raco*icf I latlon 

Total Post-^econcl I latlon 
Itaa fm|>act 
Application Error 
FIfial Item Error 
Institution Error 

Total Pro-reconci f latlon 

Tota I Pos t'-raconci 1 1 at i on 

Categorical Error 
Pra^aconcf I tat Ion 

Post-raconcI II at Ion 
Olsbursamant Error 
Pra-reconcf 1 1 at Ion 

Post-racoocI 1 1 atlon 



Last 



STlDEffTS/ 
PARENTS 



DATA SOU%£S 

INSTITUTIONS 



verif lod DIstHrsamant 
Data SAR 



X 
X 
X 
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X (Disb) 



X (SAI) 



X Cost 
Enrol Iwent 
Cattsgorlcal 
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Enrol Iment 
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Enrol Ifsant 
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X (Ofsb) 



X (SAI) 



X Cost 
Enrol Iment 



X (SAD X DIstxjrsement 
Cost 

Enrol Iment 

X Disbursement X (Dlsb) 
Cost 

Enrol Iment 



X Disbursement 
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X Categorical X (Dlsb) 
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X Dfsbt^sement X Disbursement 
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Enrol Iment Enrol Iment 
Categorical 



FIGURE 2-4 

MATRIX OF OUTCOl« MEASURES AND DATA SOURCES 
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AUXILIARY DATA COLLECTION REQUIREMENTS 



Two challenges to the validity of seunple aurvays are: 



Experimental Bias 



Nonresponse Bias 



There are two aspects of experimental bias Which might 
affect survey results givai a two-stage (school-student) sample 
selection. First, schools selected in the first stage may alter 
their behavior such that their error rates are improved when 
compared to schools not selected. Second # selected students may 
likewise behave differently from nonselected students. 

Behavioral differences, or experimental bias, were measured 
in the Stage One study as differences in corrections behavior. 
If students were reacting to selection, one vwuld expect fewer 
corrections which increased eligibility and/ or more corrections 
which decreased eligibility. Similarly, one would expect 
selected institutions to encourage this type of student behavior. 
While it is not possible to measure the extent of experimental 
bias, one can assess its existence by contrasting the corrections 
behavior of selected students at selected institutions with all 
students at selected institutions arxi by contracting corrections 
behavior of students at selected schools with students at non- 
selected schools. 

These contrasts require drawing two additional samples : 
e Students at nonselected institutions 
e Nonsampled students at selected institutions 
Additional data collected for these two samples (and for the 
sampled students at selected institutions) would involve the 
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central processor's computed applicant file* This data base 
would be used to develop profiles of corrections behavior for the 
three groups^ The profiles would encompass the number of correc- 
tions and eligibility consequences of these corrections. 

The second validity challenge i nonresponse bias^ can be 
addressed in the same way it was for the Stage One study, unlike 
most surveys, the Pell QC study allows the gathering of consider- 
able information for nonrespondents . This would include income, 
wealth and family demographic data from the application/ payment 
or award amounts, corrections behavior, and institutional data« 

Profiles of respondents and nonrespondents along these 
dimensions can be compared in order to assess the degree to which 
respondents differ from nonrespondents. 

If the differences are significant, and the response rates 
differ across groups, the profiles can be used to make detailed 
adjustments for nonresponse bias- In addition to these correc- 
tions, sensitivity analysis can be performed to assess the 
impacts of the various assumptions which might be made concerning 
error level s for nonrespondents • 



CHAPTER 3 

DATA COLLECTION AND SAMPLING ALTERNATIVES 

Once decisions have been reached about a definition of pay- 
ment error # the components of that error to be measured, and the 
types of data required for such measurement, the following issues 
must be considered: 

• Data collection alternatives (What is the best way to 
obtain the desired error measures?) 

• Sample selection alternatives (What is the best method 
for selecting a nationally representative sample of 
recipients?) 

• Sample size implications (What are the required sample 
sizes , given various levels of confidence and 
precision?) 

DATA COLLECTION ALTERNATIVES 

At a minimum^ the following information must be collected on 
each sampled recipient in order to measure total Pell Grant 
error: 

1» Actual Pell Grant disbursement 
2* SAI used to calculate disbursement 
3« Cost of Attendance 
4* Enrollment Status 

5* Documentation that verifies student's categorical 
eligibility for the Pell Grant 

6* Documentation that verifies application data 

Required data items one through five — disbursement* SAI, cost, 

enrollment, and eligibility — must be collected from student 
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records at institutions^ and required item six — documentation 
that verifiefi application data — must be collected directly from 
the student and parent atnd/or from agencies and organizations 
such as the IRS, tax assessors, and so on* 

Four discrete approaches for collecting the required infor- 
mation can be identified. The four data collection options are 
distinguished by the method which data are collected: in- 
person data collection, telephone interviewing, and mail survey. 

These options are presented in Figure 3*1 « In option 1, 
student record data (required items one through four) would be 
collected by site vidits to institutions, and student /parent data 
(required item six) would be collected by in-person interviews 
and by mail from the IRS and possibly from other agencies and 
organizations • Option 2 would have the same features as Option 1 
except that in-person interviews with students and parents would 
be replaced by telephone interviews. Option 3 would be a scaled- 
down version of Options 1 and 2: no student and parent inter- 
views would be conducted. Option 4 would have no field work; all 
institutional and student /parent data would be collected by mail- 

In Figure 3--2 the four options are compared according to 
their ability to meet the following objectives for an ongoing 
Pell Grant error assessment systems 

• Minimize Financial Cost 

# Maximize Reliability of Error Measures 

^This assu:aes a pre-reconciliation measure of error { see 
Figure 2-4). For a post-reconciliation measure the disbursement 
SAX would be taken from the PIMS reconciliation roster. 



OPTIOW 1 



OPTIOH 2 



(FTION 3 



OPTIWM 



Institutfonai Oala 
Collection 



Visits to Institu- 
tions 



Visits to Institu- 
tions 



Visits to Institu- 
tions 



Collection of Institu- 
tion data (actual dis- 
bursemnt, cmrollmnt 
status, SAl, cost) by 
mail from Institution 



Stu{*ont /Parent 
fVlri«ry Data 
Collection 



in-^son Interviews 
with students and 
parents 



Telephone Interviews 
with students and 
pwents 



None 



None 



Student/Parent 
Secondary Data 
Col lection 



Collection of release 
forms from students or 
parents and collection 
of verifying docunm- 
tat Ion by mail from 
agency/or ga n I zat I onv 



Collection of release 
forms from students or 
parents and collection 
of v^lfylng documen- 
tation by mall from 
agency /or ^nl zat Ion 



Collection of release 
forms from stu<fents or 
parents and coll€K:t|on 
of vwlfying documen- 
tation by mal 1 from 
agency /organ I zat I on 



Collection of release 
forms from students or 
parents and collection 
of verifying documen- 
tation by mal I from 
agency/organ I zat Ion 
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OPTION 1 



OPTION 2 



OPTION 3 



OPTION 4 



Object lyes: 



Hlnlmlze Federal 

Cost 



Host costly optloo* 
High costs ossocleted 
with fr9^* eiKl tim 
spent covHluctlng In- 
person sht^iMf/pm'enf 
Intervlairs* In con- 
trast to telephone 
IntervlM^s^ In-person 
IntarvieM generally 
require aore Know' 
ledgeable and reli- 
able Intervienera* 



less costly than 
Option U 

Costs associated with 
telephone Intervlev- 
Ing (person hours an < 
telephone bills) are 
less than costs asso- 
ciated with In-per^son 
Interviewing (person- 
hours, fransfiorfa" 
Hon, lodging). 



Less costly than Ca- 
tion t and Option 2. 
With no student/ 
parent interviews^ 
iBjor costs arise 
from tine and travel 
for Instltutlofial 
visits* 



Least costly option* 
Ho travel costs* 
Gmerally# aail 
surveys can rely on 
lower salaried cleri- 
cal staff, whereas, 
field dita collection 
requires more Know* 
ledgeable and mnper" 
lenced personnel* 



MdHtaize Reliability 
of Error Measures 



I 



High reliability since 
verifying docmmita- 
tloft on all 24 appli- 
cation i teas used to 
calculate SAI col lect- 
ed during in-i>erson 
Intwviews, Institu- 
t.lonai visits, and 
from secondary sources 



tower reliability than 
Option t since, with- 
out In-perscn inter- 
views, considerably 
less documentation 
collected* Idould have 
to rely on a limited 
student error defini- 
tion, a*g*, student 
error based on tax 
r^ii^ft error 



(Same as Option 2) 



Low Raliabllity* 
Mould hav« to rely on 
Halted student error 
deflnltlofi* Dmta 
received by mail fros 
Institutions would be> 
unvar if led« 



Maximize Precision 
of Error Estimates 



Sample sUe is a 
i^iermtnmif of accur- 
acy (see Figure 3-3A- 
3-50)* To reduce 
travel costs, recip- 
ient sample would have 



Essantlaiiy same as 
Option t* However, 
with no in-person 
student/oar erf inter- 
views, the f\$94 to 
cluster institutions 



(Same as Option 2) 



Hith no travolling, 
no clustering needed* 
This would eliminate 
the undesirable 
design effects of 
cluster sas^Mng* 
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FIGURE 3-2 

SUNMARr OCM^AHiSON OF DATA COI tECTION OPTIONS 
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OPTION 1 



OPTION 2 



OPTIW 3 



OPTION 4 



Maxlwiza fVeclslcxi 
of Error EstlMtos 
(Continued) 



to be cfttst«rod« 
Cluftorlfig tends to 
roduoa accuracy of 



geographically to 
reduoa travel cc»ts 
fiot as great* Note: 
Given a set budget, a 
larger sj«i«»le could be 
dira«m under this op^ 
tlon than uodor Option 
1 since plione Inter- 
viewing Is less costly 
than In'-porsoo Inter- 
vieitfing. 



Hotet Given a set 
budget, e much larger 
saaipfe could bo drown 
under tbis option 
than under any other 
option since data 
collection costs are 
Idlest «, 



NaxlAlxe Response 
Rate 



IaI 

I 



Relatively high 
response rate 
I n-per son student/ 
perenf IntervlcMrlng 



Response would tehd 
to be low^ than In 
Option I, Some 
students would not 
have telephones* 



Response would tend 
to be lower than In 
Options 1 and 2, un- 
less the nai I survey 
were followed up 
aggressively by 
telephone cal Is* 



(Same as Option 5) 



Ensure Col lection 
of Ancil lary gC 



Oata 



Could be collected 
through Interviews 
and f I le reviews 
at Institutions and 
Interviews with 
s tu dents/ parents 



(Smo as Option f) 



Could be col lected 
froM institutions, 
but not froHi 
s tu dents/parents 



Unless this option 
Included a Ml I Sur- 
vey of institutions 
and parents, It could 
not be col lected* 



^Ancillary QC 4afa include data not needed to masure error ^ For axaeple, 6afa collected to ascertain the reasons for paymnf arrcyr 
are considered ancillary QC data* 
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Mini«ii0 BuTiJ^n on 



Minimize Tlroa-^raflie 
for Udtd Col lection 



OPTION ! 

Tim burden on \(^^ 
stltutional person-* 
nel: 1-6 hoirs« TIm 
burden on stu<lents/- 
parentss 2^ hours* 

Depending on saople 
size, saoipllng nethod, 
staff size, and other 
factors « nonths 
rBqulred to select 
sample and coisplete 
col lection of data 



OPTION 2 



(Sagie as Option t) 



I 



Shorter period of tins 
than Option 1: 3-5 
(Months depending on 
SMple slze^ sMplIng 
oiethod, staff size, 
etc* Unanticipated 
problems such as 
lndleq»nt weather do 
not ^l^upf phone 
Interviewing schedule 
as they nl^t In- 
person Interviewing 
sctodule* 



ION 3 



Institution bM*den 
saiee as (^tlon I 
student/parent Utf-den 
much less than In 
Options 1 and 2 

(Smb as (H)tion 2) 



OPTION 4 

Institution bidden 
potentially greater 
than In Options f-3« 
Student/parent burden 
same as Option 3* 

Sorter period of 
time than Options 
l-3i m estimated 2-9 
months depcHidlng on 
staff size, sample 
size and other 
factors 



Hdxlmiie Feaslbl llty 
of Developing ED 
In-tiouse CapabI 1 1 ty 



With training and 
specif leaf Ions, insti- 
tutional visits could 
be conducted by DCPR 
program reviewers* 
Considerable costts 
would be associate 
with developing 
nationwide student/ 
p^ent I nter vi ew i ng 
capabI tity. 
Minor costs would 
be associated with 
developing mall survey 
capabI 1 1 ty 



Essentially same as 
Option I, however, 
development costs for 
phone Interviewing 
would be less fi\an for 
developing In-person 
interviewing 
capabi llty 



With no student /parent 
Int^vlMs, develop- 
ment costs would be 
relatively minor 



(Same as Option 3) 
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SiM^Y CQHPARISON OF DATA COCLECTION OPTIONS 



m Maximize Precision of Error Estimates 

• Maximize Response Rate 

• Ensure Collection of Ancillary QC Data 

• Minimize Burden on Respondents 

• Minimize Timeframe for Data Collection 

• Maximize Feasibility of Developing ED In-House 
Capability 

Option 1 

This option would l>e essentially the same as the approach 
used in the Stage One study and wuld include visits to insti- 
tutions, in-person student /parent interviewing, and data collec- 
tion by mail# It would be the most costly of the four options 
since there are major costs associated with nationwide in-person 
interviewing « These costs include travel, lodging, salaries, and 
per diem for field wrkers as well as salaries for support staff. 
This option also would require the longest period of time to 
implement* Even with a small sample, a minimum of four months 
would be required to select the sample and complete the data 
collection. 

ED'S Division of Certification and Program Review (DCPR) 
progreun review staff, with training and specifications, could 
conduct the institutional site visit component of this option • 
However, it is very unlikely that ED, in the short-terra, could 
develop an in--house capability to conduct in-person student and 
parent interviews ^ Interviewing a nationwide sample of parents 
and students requires a nationwide network of interviewers such 
as Westat used in the Stage One study. 



In spite of these drawbacks, this option, of the four, would 
provide the most reliable error measures and the most useful 
information for corrective action decision making. As discussed 
in Chapter 2, the reliability of the error measures depends in 
large part on the extensiveness and intensiveness of the data 
collection. For example, to measure student error properly. 
Verifying docuii.jntation of all 24 application items used to 

/ ^^^^^^ 

calculate the SAI must be collected for each recipient in the 
sample. Interviewing students and parents in person, winlike 
interviewing by telephone or not interviewing at all, would 
ensure collection of a substantial amount of valid verifying 
documentation of application items. Of the four options, this 
option is the roost extensive and intensive and thus would give 
the most accurate reading of program error. 
Option 2 

In this option, students and parents would be interviewed 
over the telephone rather than in person. There are several 
operational advantages to interviewing by telephone. First are 
cost considerations. Sc^ne telephone interviews are similar in 
cost to personal interviews if the survey requires long distance 
dialing between nine to five and involves a lengthy interview 
and/or requires talking with a number of people before the target 
respondent can be identified. However, these situations would 
probably not apply to a Pell Grant error study and therefore it 
is likely that, in this case, the telephone interview would coat 
less than the in-person interview. Second, telephone inter- 
viewing can be conducted in a much shorter period of time than 
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in-p€rson interviewing. Finally, unanticipated problems such as 
staff attrition are controllable if the survey is a telephone 
survey instead of a personal interview since problems can be 
immediately identified and solutions can be quickly implemented. 

There are disadvantages to interviewing by telephone. 
First, many respondents would not have telephones or could not be 
reached easily by telephone. Second, and more important, tele- 
phone interviewing would require that the study rely heavily on 
obtaining documents from a secondary data source (i.e, IRS, tax 
assessments, bank, public assistance offices, etc.) since in- 
person visits to view verifying documentation would not be con- 
ducted. Therefore, error measures under this option would be 
less reliable than under Option 1 since they would more than 
likely be based on an incomplete set of data. 
Option 3 

In this option, students and i ents would not be inter- 
viewed • This option has the same strengths and drawbacks as 
Option 1, the major strength being its relatively low cost and 
the major drawback being that all verifying documentation of 
student application information would come from se^condary 
sources. This option has one additional drawback. With no 
student and parent interviews, data that could be valuable to 
corrective action decision making would not be gathered- For 
exeunple, in the Stage One study, valuable insights into the 
sources of student payment error were obtained during the student 
and parent interviews. 
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Op-tion 4 

In Options 1, 2, artd 3 institution data would be collected 
through site visits. In Option 4, however, the required insti- 
tution data would be collected by mail by providing each sampled 
institution a set of detailed specifications for abstracting 
information from the sas^led recipients' files. This option has 
several operational advantages over the others. First, with no 
field work, it could be conducted at a relatively low cost. 
Second, the recipient sample would not need to be clustered, thus 
eliminating the undesirable design effects of cluster sampling. 
Third, ED in the short term, with relatively little investment in 
new staff/ could develop an in-house capability. There are two 
serious drawbacks in collecting institutional data by mail. 
First, the burden on institutions could be significantly greater 
than in the other options . Second , and more important , data 
rv«ceived from institutions on saunpled recipients would be unveri- 
fied. .This would detract significantly from the reliability of 
the error estimates. 
Recommendation 

This ass -ssment of tlie four options indicates that Option 1 
is the desirable data collection approach for an annual assess- 
ment of Pell Grant error. It has by far the most potential of 
the four for providing accurate and reliable error measures . In 
Chapter 4, technical specifications for this approach are set 
forth in detail . 
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SAMPLE SELECTION ALTERNATIVES 

Although the principle determinant of the accuracy of infer- 
ences based on a sample is the number of cases included in the 
sample r the way in which a sample is selected also affects 
accurfacy. Clustered san?>les will usually less accurate than 
unclustered ones because the clustering process itself is" a 
potential source of error . 

Therefore, the most accurate estimates would be based on a 
sample of all Pell Grant recipients^ in which each recipient's 
chance of being included would be eq»ial, not dependent on asso- 
ciation with a cluster r such as attendance at a particular insti- 
tution. Such a sample could be drawn at any time from the PIMS 
recipient file* A simple random sample of those whose SAR data 
have been entered as of a certain date could be of any size ^ 

This sampling method has two disadvantages, however- Since 
it is not clustered, practically every medium or large institu- 
tion, and many small institutions^ would be represented, most of 
them by only one student* Institutional data collection would 
involve visits to so many institutions as to be completely unfea- 
sible* The students themselves would also be extremely disbursed 
geographically, which would make interviewing them also unfea^ 
sible* Therefore, an unclustered sample would only be used in 
conjunction with a data collection procedure that involves visit- 
ing neither institutions nor students — in other words. Option 4* 

If the student /parent sample is to be clustered at institu- 
tions , a two-stage sample must be drawn • First , a sample of 
institutions must be drawn from the universe of eligible 
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institutions; then a seunple must be drawn of the recipierts 
attending those institutions. One way to do this is to secure a 
list of each institution's recipients and sample from each list 
at the t'tudy's main office. That is the procedure specified in 
Chapter 4» The JuSvantages of the procedure are that the sampling 
process is entirely under the control of the QC project staff, 
and the sample can be drawn before the institution site visits 
are made. However, there is a burden on the institutions, which 
must list all their Pell recipients by name with their grant 
amounts, addresses, telephone numbers, and Social Security num- 
bers; and a considerable administrative burden on the praject 
office, which must compile all the lists, pester laggard insti- 
tutions, and draw all the samples. 

An alternative method of drawing a clustered sample is to 
leave it up to the site visitors. They would be trained in the 
sampling n athod to be used and told how many recipients to sample 
from each institution. Upon arriving at the institution, the 
site visitor would draw the sample from a list of the recipients 
prepared by the school. 

This method was used during the fall, 1982 validation evalu- 
ation to draw a sample to be used then and for the spring, 1983 
data collection for Phase Three of the Quality Control Project. 
The overall sample was allocated among institutions based on 
measures of size from the previous year (as would probably have 
been done no matter how the sample was drawn). A systematic 
selection interval was calculated for each institution which 
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produced the desired number of respondents # and then a randcm 
start number was assigned. Prc»n the interval and start number, a 
sampling worksheet was composed which listed the sequence numbers 
of all the recipients to be sampled from each institution, 
assuming that the recipients could be numbered consecutively 
starting with **1** at each school in the sample. 

w 3b 1 eras arose with this method \^*ien the field staff tried 
to get lists of recipients at the institutions^ Because the 
sampling method had been worked out too late for the list to be 
requested in the initial letter to the institutions, each insti- 
tution was asked during the scheduling telephone call to compile 
a current list of paid Pell recipients and have it ready for the 
data collector on arrival. However, the instructions to the 
telephone staff were not extensive enough to cover some unusual 
situations; sometimes explicit instructions about what the list 
should contain could not be given over the telephone- Moreover, 
many institutions either could not compile a list which would 
meet the project requirements, or were unwilling to take the time 
which would have been required. 

Therefor the site visitors were confronted with a great 
variety of lists and, at many institutions, no list at all. 
Frequently, the list which was avaixable included people who were 
not part of our study, such as recipients or aid other than Pell 
Grants, or ot aid applicants who had presented valid SARs but 
had, finally, not enrolled in the institutions. The data collec- 
tors had to examine such lists and delete the irrelevant entries. 
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At some institutions, this could be done only by consulting other 
records or lists Which ware not in the same order. In those 
cases, it was necessary to take a preliminary sample with the 
knowledge that it included people who were not subjects of the 
study and would, therefore, be longer than desired- Only the 
records of these persons would be checked to delete the ineli- 
gibles and arrive at a valid sample. If data institutions had 
not compiled lists, collecters were to do this manually in order 
to document fully the sampling procedure. 

Each of these different procedures was often very time con- 
suming. Compiling lists where none existed was quickly found to 
be impracticable at all but the smallest institutions. Data ^ 
collectors counted through stacks of SARs, drawers of file 
folders, or boxes of file cards to draw samples. Frcan a planning 
perspective, the most severe problem arose when data collectors 
had to compile or edit lists. It was impossible to tell in 
advance if this was the situation at a given institution. At 
some institutions, compiling the list and drawing the sample took 
all the time scheduled for the entire data collection. Data col- 
lectors were forced to stay an extra day at some of these insti- 
tutions, in which case other parts of their schedules were 
rearranged or a special "floater" was sent in to complete the 
scheduled visits - 

Another problem encountered in drawing the sample in the 
field was caused by multi-campus institutions with decentralized 
record systems (each campus keeping all the records for its 



recipients) that filed a single Progress Report. S^nce Progress 
Reports were used to draw the sample of institutions ^ it was not 
possible to draw a single campus in advance. When scheduling 
these visits it was also difficult to determine whether the 
required information was kept at the main campus or the branch 
campus. Thusr in son» instances data collectors arrived at a 
campus only to find that the records of some students were else- 
where, even in another state* Subsaropling procedures , vhich 
involved getting a count of the number of recipients or students 
at each campus and consulting with the project office by tele- 
phone, had to be developed after the data collectors had gone 
into the field. In one case this resulted in the selection of a 
campus in another state. The financial aid administrator had to 
fly to the site and the "•floater** data collector and a member of 
the central project staff had to be sent in to collect the data. 
In other cases problems arose when the branch campus selected 
was not expecting a visit because the notification letter had 
bean sent to the main campus. 

Timing is another problem with this method. The site visits 
roust be completed before initiating, student/parent data collec- 
tion. However, the institution visits cannot be made until late 
in the spring (April and May), after final payments have been 
posted to student records (otherwise, one would not know how much 
money a student actually receive<f) . Obtaining the student sample 
in April or May v uld require interviewing the students in the 
summer. That would be a year or more after many of them had 
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filled out their Pell applications. Moreover, many of the 
addresses on the institution's records and on the SAR would no 
longer be valid, requiring a major tracing effort and a large 
increase in the number of unlocatable students. (Even if the 
parent, address did not change, some students %»rould be spending at 
least part of the summer elsewhere, and most graduating seniors 
v^uld have found jobs and established their own households.) 

The only readily Apparent solution to this problem, other 
than securing sample lists by mail as recommended, is to send 
specially trained samplers to the institutions in the fall. In 
effect, this is what was done in the fall of 1982, however, the 
sampling was combined with a special data collection done to 
evaluate validation procedures. To send people out only to col- 
lect the sample would be too expensive when canpared to the 
recommendad method. 

SAMPLE SIZE IMPLICATIONS 

The number of students who must be sampled depends on four 
factors! the degree .of precision wanted for the statistical 
inferences derived from the data,- the amount of confidence 
desired in these estimates (the significance level); the degree 
of clustering to be used,- and the degree of homogeneity within 
each cluster. Figures 3-3A through 3-3D at the end of this chap- 
ter show the number of recipients needed to achieve some common 
precision and confidence levels assuming specified cluster sizes 
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and degrees of intracluster homogeneity « The table entries are 

calculated from the formula 

V2 ^ (l-pA / \ / ta 

e 1+ P (nc-1 ) f 

"--p 1 \ I I half width 

where: 

n « the number of cases required 

p m proportion of cases with error 

v2 a the relative variance of the means of the cases 
® with error 

p = the intracluster correlation 

nc » the number of cases in each cluster, i.e., the 
number of students at each sampled institution 

t ^ the standard normal deviate associated with a 

particular confidence interval 

The first two tables are based on the assumption that intraclass 

correlation is quite low ( P «■ .1), that is, that the students 

at each institution differ considerably eunong themselves. In the 

last two tables, a higher intraclass correlation (P » .5) is 

assumed, that is, there is significantly less variation among the 

students at each institution. In Figures 3-3A and 3-3C it is 

assumed that an average of only 5 students are selected fr<»n each 

sample institution? in Figures 3-3B and 3-3D it is asstuned that 

15 students are selected from each sample institution. 

Within each table, four precision levels and four confidence 

levels are given. The precision level is how close the estimate 

made from the sample must be to the true value. Which one could 

determine only by collecting data from every recipient or his > 



3-17 



ERIC 



66 



* 



parents* Th«9e precision l«vels ars expresssd as proportions of 
the sample mean. For instance/ if one had an average error of 
$100/ a precision level of .10 would indicate that the actual 
mean is $100 + .10 ($100)/ or between $90 and $110; a precision 
level of .025/ on the other hand, would indicate that the mean is 
$100 + .025 ($100)/ or between $97.50 ard $102.50. The precision 
level is also called a "half width" ^ecat:.se it is half as wide as 
the range within which the population value probably falls 
(because it is both added and subtracted to the sample value). 
The confidence level indicates how sure one can be that the popu- 
lation value actually falls within this range, expressed as a 
probability. Thus, a confidence level of .975 indicates that 
there is a 97.5 percent chance that the population value is equal 
to the sample value plus or minus the precision level? if 200 
different estimates were made at this confidence level/ only 5 of 
them would differ from the true population value by more than the 
pr-acision level. At the lowest confidence level given, .90, I in 
10 of the estimates would be off by more than the precision 
level. 

Within each table it is obvious that the greater the preci- 
sion desired (expressed by smaller half-widths), the largver the 
sample must be; likewise; the higher the confidence level 
desired, the larger the sample must be. These two requirements 
are cumulative, so that if both greater precision and higher con- 
fidence are desired, the sample must be very large indeed (note 
the lowest left-hand box in each table). 
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Clust«r size and intracluster correlation also affect the 
required sample size. Larger cluster sizes make both the insti- 
tutional and the student/parent data collection more efficient if 
visits are involved, but there is a loss in accuracy, expressed 
in Figures 3-3B and 3-3D by larger required sample sizes for the 
same precision and confidence levels. Higher levels of homogen- 
eity amdng the students in each institution (expressed by higher 
values of p ) require larger samples for the same accuracy. As 
with precision and confidence levels, the effects of cluster size 
\an(^^intracluster homogeneity are cumulative, so that both a large 
c^ust^ size and a high level of homogeneity within clusters 
req\^ires the largest samples. Since t^^ese effects are also 
cumulative with precision and confidence effects / the largest 
saunple ifi the Figures is that required for high precision 
(half-width = + .025), a high confidence l6vel (confidence level 
= .975), a large cluster size (nc =^ 15), and a high intraclass 
correlation (p » -5); this is the entry in the bottom left cell 
of Figure 3-3D, a sample of 64,225. 

For the purpose of calculating sampling error, a subsample 
can be treated as an independent sample ont a smaller popula- 
tion^ The principal determinant of the accuracy of any sample or 
subsaunple io the sheer size of the sample; the proportion the 
sample bears to the population is largely irrelevant unless the 
proportion is very large (generally on the order of 20 percent or 
more), much larger th^n would be practicable for the Quality Con-- 
crol Project. Since any subsample must be smaller than the 
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sample of Which it is a part, estimates based on subsamples will 
be less accurate than estimates based on the entire sample except 
in highly unusual situations not likely to be obtained for the 
Quality Control Project. In other words, conclusions about 
subgroups of Pe] 1 Grant recipients not filing 1040s or 1040AS, 
must be less certain than conclusions about all Pell recipients. 

For example, if the cluster size were limited to 5, and the 
intraclass correlation was low (P =» .1) — as shown in Figure 
3-3A — a sample of 2,397 would be large enough for 95 percent con- 
fidence and a precision level of plus or minus 5 percent. Sup- 
pose, however, that 2 5 percent of all recipients did not file an 
IRS return. A sample of 2,397 would include about 599 such 
people. But Figure 3-3A shows that with only 599 cases, one 
could be only 95 percent confident in a precision level of plus 
or minus 10 percent of the sample value. In other words, esti- 
mates for non-filers alone would only be about half as accurate 
as estimates for all students (including non-filers). 

This phenomenon also applies to groups which are smaller 
than the full sample because of inapplicable or missing da^a, or 
to conclusions about causes of error which are based only on 
recipients with that error rather than on all recipients- For 
instance, only 23 percent of all recipients in the 1980-81 study 
sample had hard documentation of home debt. Error estimates for 
this group '^lone would be only about half as accurate as esti- 
mates for all recipients. For some' application items, the pro- 
portion with documentation was only about one percent. Error 
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estimates for these groups, if based only on recipients with 
documentation, would be only about one-tenth as accurate as esti- 
mates based on the full sample • 

In summary, follow these three steps to calculate the 
accuracy of making estimates about a subgroup or subsample* 

1. Estimate the size of the subsample, (For example, if 
one knows that the subsample is 50 percent of the full 
sample and the size of the full sample is 1,000 the 
estimated size of the subsample is 500.) 

2. Determine the cluster size and interclass correlation. 

3* Locate the subsample size on the appropriate table 
(Figures 3-3A and 3-3D) and find the confidence and 
precision levels* 




ERIC 



Half -Width 




Confidence 


Level 




or 

Precision 


.975 


.950 


.925 


.900 


+ .10 


783 


599 


494 


425 


+ .075 


1,392 


1,065 


879 


755 


+ .050 


3,131 


2,397 


1,977 


1,699 


t .025 


12,524 


9,589 


7,908 


6,795 



FIGURE 3-3A 

PRELIMINARY ESTIMATES OF 
REQUIRED RECIPIENT SAMPLE 
SIZES ASSUMING THAT 
p « . 1 
=« 5 
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Level 
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or 

Precision 
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! . 900 
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+ .10 


1,340 


1,026 


846 


727 


± .075 


2,382 


1,823 


1,504 


1,292 


± -050 


5,359 


4,103 


3.384 ^ 


2,908 


+ .025 


21,435 


16,411 


13,535 


11,631 



FIGURE 3~3B 

PRELIMINARY ESTIMATES OF 
REQUIRED RECIPIENT SAMPLE 
SIZES ASSUMING THAT 

P - .1 

nc = 15 
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Level 




or 

Precision 
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1,283 


1,058 
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+ .075 


2,979 


2,281 


1,881 


1,617 


+ .050 


6,704 


5,132 


4,233 
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+ .025 


26,814 


20, 530 


16,932 


14, 549 



FIGURE 3-3C 

PRELIMINARY ESTIMATES OF 
REQUIRED RECIPIENT SAMPLE 
SIZES ASSUMING THAT 
p «» . 5 
He » 5 
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Confidence 


Level 




or 

Precision 


.975 


.950 


.925 


.900 


+ .10 


4,014 


3,073 


2,535 


2,178 


+ .075 


7, 136 


5,464 


4, 506 


3,872 


± '050 


16,056 


12,293 


10, 139 


8,712 


+ .025 


64, 225 


49,172 


40, 556 


34,848 



FIGURE 3-3D 

PRELIMINARY ESTIMATES OF 
REQUIRED RECIPIENT SAMPLE 
SIZES ASSUMING THAT 
P - -5 
Uq ™ 15 
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CHAPTER 4 
DATA COLLECTION SPECIFICATIONS 



OVERVIEW 

In Chapter 3 it was reconunended that ED collect data for an 
ongoing annual assessment of error using a three- faceted 
approach: visits ±o institutions to collect information from 
student aid files / field interviews of Pell Grant recipients and 
their parents, and collection of verifying documentation by mail 
from organizations such as the IRS, banks, public assistance 
offices, and tax assessors. In addition, it was recommended that 
the Pell Grant recipi^ant sample be selected by project office 
staff in advance of the field work from up-to-date recipient 
lists requested from sampled institutions rather than by field 
representatives during the institutional visits . 

Specifications for collecting data using this approach are 
set forth in detail in this chapter. In this particular section, 
several important assumptions made in developing these specifica- 
tions are listed. Also in this section is a discussion of 
several issues that Iraust be considered when scheduling the data 
collection. In the remainder of the chapter, technical 
specifications for collecting data are presented task by task and 
step by step. Four general tasks are identified; sample selec- 
tion (TASK 1)/ student and parent interviews (TASK 2), collection 
of secondary data, i.e., verifying dociunentation from the IRS and 
other organizations (TASK 3), and visits to institutions (TASK 
4). Each task has been divided into subtasks . The descriptions 
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of each subtask follow a set foxrmat; first, a discussion of the 
purpose of the subtask and any important issues that need con- 
sideration and* second/ a step-by-step list of procedures. 

Figure 4-1 presents an overview of the tasks and subtasks 
described in this chapter in a way that illustrates their inter- - 
dependencies. No time line has been illustrated in this charts 
nor should one be. implied; its express purpose is to describe the 
path by which data collection must proceed. 

Major Assumptions Made in Developing Data Collection Specifi - 
cations 

The technical specifications as set forth in this chapter 
make no presupposition with regard to sample size^ the organi- 
zation(s) responsible for collecting the data, or the time of 
year and duration of the data collection. In other words, the 
procedures specified hold true regardless if the recipient sample 
is 50C or 5,000. if the field work is conducted by regional 
program reviewers from the DCPR or by a contractor, or if the 
data are collected in one month or six months - 

However, it was necessary to make several important assump- 
tions about ED' 3 goals for an ongoing assesment and about Pell 
Grant program changes when developing the data collection specie 
fications. The following were assumed: 

# ED will wish to measure total payment error as well as 
its subcomponents, institution error and student error 
and their subcomponents, categorical error, calcula- 
tion/accounting error, and application item error (see 
Figute 2-2, ''Decomposition of Total Payment Error''). 

• i::D will wish to collect data not directly needed to 
measure payment error. For example, ED will wish to 
collect da^a on institutional procedures as part of the 
institutiortal site visits . 
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# ED will not wish to collect data on recipients who 
reside in Alaska, Hawaii, Puerto Rico, the Virgin 
Islands, and other U*S* territories. Field work will 
be confirmed to the 48 continental states* 

• Recipients whose applications are selected by the Pell 
Grant processor for validation will not need to be 
oversampled, as was done in Stage One, in order to have 
a large enough sample to support accurate inferences 
about all validated students • From now on, all, or a 
large proportion, of the Pell population will be 
selected for validation* 

Schedule Considerations and Constraints 

As discussed in Chapter 2, the timing of the data collection 
can have an important impact on the ability of students, parents, 
and institutions to provide accurate data. The Pell Grant cycle 
extends' from January when students begin to fill out application 
forms to over 15 months later in July when institutions begin to 
reconcile and close out their Pell Grant accounts. An institu-- 
tion sample can be drawn at any time during this cycle- However, 
the earliest point at which a recipient sample can be drawn is 
early to mid-fall, after institutions have calculated and posted 
their initial Pell Grant disbursements. 

Ideally, from a design perspective, the data should be col- 
lected frcmi students and parents immediately after s^u^ple selec- 
tion. Students normally apply for Pell Grants in winter or 
spring preceding the academic year for which they are applying 
for aid. There is evidence that the longer after the application 
date that one waits to collect data the less reliable are the 
recall of the student and parent and the documentation provided. 
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There are reasons/ howc-er Why it may neither be feasible nor 
desirable to begin student and parent interviews and collection 
of secondary data in, for exaunple, November after an October 
sample selection. First, selection of the recipient sample in 
the manner specified in this chapter may take 8 to 10 weeks. 
Lists of Pell Grant recipients must be requested and received 
from each sampled institution before a recipient sample can bo 
picked and field work begun. Before secondary data can be 
collected* SARs of the sampled recipients must be requested a.id 
received from the scimpled institutions. Even if a recipient 
sample could be selected quickly (i.e., by the beginning of 
November), student and parent interviews should not be scheduled 
for November and December because of the disruptions caused by 
the holiday season. Many students during this time are away from 
school or traveling and would be difficult to locate. Therefore, 
the optimal time to be9in interviewing, both in practical and 
study design terras, would be January. 

There is no ideal time foi: collecting secondary data, 
although obtaining documentation by mail may take longer during 
the holiday season, the busiest time for the Postal Service, and 
from January to April, the busiest time for the IRS. 

Another important issue is the duration of the student and 
parent field work. Extending the field period over several 
months could have research design implications. Students and 
parents interviewed in January would have different character- 
istics than those interviewed in July. Those in the January 
group, for example, would tend to provide more reliable data than 
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thosa in the July group. A very complicated weighting scheme 
wouia have to he designed to allow for this design effect. 
Ideally, then, the field work should be completed in the shortest 
time possible. 

While the student and parent interviews should be conducted 

in the winter, the institution site visits must b^ conducted in 

mid to late Spring- Information from student financial aid 

records cannot be abstracted until after institutions have posted 

the final Pell Grant disbursements * of the year. For institutions 

on the semester system, posting normally occurs in January or 

February* However, for institutions on the quarter system and 

for many schools that use a clock hour system, it does not occur 
♦ 

until March or April. The latest point at which institutional 
data should be collected is July, the beginning of the next Pell 
Grant award year* After that point, institutions have reconciled 
their students' accounts, and it would be very difficult for a 
data collector to discern "true" from reported data in a 
student's record* For example, it may be difficult to determine 
if statements of academic purpose or financial aid transcripts 
have been collected if data collection is conducted late in the 
cycle, since these records may no longer be easily accessible / 
particulai'ly at institutions with highly ai^tomated systems* 

TASK 1; SELECT SAMPLE 

Information on students must be drawn from institutional 

files, and data on each student's institutional content must also 

be gathered during a site visit* Therefore, it is neces'sary to 
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cluster the students by institution and sample only the students 
from a select number of institutions. Although this group of 
institutions will be referred to as the "institution sample," 
those institutions will not, in fact, constitute a sample from 
which conclusions can be drawn about all such institutions in the 
United States. It will not be possible to estimate what propor- 
tion of all institutions have a certain characteristic or What 
the average institution's value is on any variable..^ It will be 
'possible to say what proportion of students attend institutions 
with a certain character is ic or what the institutional score is 
on some variable for the average student. For example, it will 
not be possible to estimate what proportion of ail institutions 
Vequire a C average as evidence of satisfactory academic pro- 
gress, but it will be possible to say what proportion of students 
attend schools with such a requirement; it will not be possible 
to say how many recipients attend the average institution, but It 
will be possible to say how many recipients attend the average 
recipient's insrtitution. 

Clustering, necessary for efficient site visit logistics, 
requires that the institution sample be drawn first. 
Task l.li Select Institution Sample 
Procedures 

1. Determine the Sampling Frame . A sampling frame is a list of 
the population being studied from which a sample can be drawn. In 
the Stage One study, the PIMS Institutional Master File was used 
as the sampling frame. This was supposed to in^ude ail 
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institutions participating in the Pell Grant program. Although 
experience indicated that some schools on the list were no longer 
participating {some were defunct), this is still probably the 
most up- to-date and comprehensive list of participating institu- 
tions . 

2* St ratify the Sampling Frame . Dividing the institutions in 
the sampling frame into several exclusive groups gives the 
researcher control over the probability oi selection of various 
types of schools. In the Stage One study, for instance, a small 
number of large institutions were pulled from the list and set 
aside to be iicluded in the study with certainty. Institutions 
in the 25 largest metropolitan areas were also treated separately 
because they have a large proportion of all Pell Grant recip- 
ients, and sending site visitors to all such areas posed no 
logistical problems. Institutions under the Alternate Disburse- 
ment System (ADS) constituted further strata. 

Although stratifying the sampte does permit the inclusion of 
some institutions with certainty, it also complicates the calcu- 
lation of sampling errors and of the drawing of the sample in 
general. It is recommended that the inclusion with certainty of 
large schools and large metropolitan areas be reconsidered. Such 
large clusters will have very high probabilities of inclusion in 
any randcOT-sample design, although not all of them will be 
included. Stratif icat.' on should not.be necessary to ensure that 
either lar institutir>ns or institutions in large metropolitan 
areas are jL^ncluded in the sample in proper proportion. For 
instance, large metropolitan institutions in the Midwest should 
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be adequately represented even if the Chicago, Detroit, Cleve- 
land, and Milwaukee metropolitan areas are npt all represented. 
Given the very large size of areas like New York and Los Angeles, 
it may be wise to make sure that they are included no matter how 
small the probability that they will fail to be selected; but the 
number of certainty metropolitan areas could be reduced. (For 
instance, is it important to ensure that Milwaukee, the 24th Q 
largest metropolitan Area, be included, but not Cincinnati, the 
26th — or that both Miami, the 23rd, and Tampa-St. Petersburg, the 
25th, be included?) 

3. Cluster the Sampling Frame * In order to reduce long- 
distance travel costs, the large number of institutions outside 
any metropolitan areas included with -certainty should be grouped 
by location so that several institutions can be visited for the 
cost of a single airplane ticket. A limited number of clusters 
should then be chosen at random. The clusters should be as 
nearly equal in total number of Pell Grant recipients as pos- 
sible- In the Stage One study, institutions were sorted by 
three-digit zip code prefix. Those areas were then grouped so 
that no cluster contained more than twice the average number of 
students or less than half; therefore, the largest cluster was 
not more than four times as large as the smallest cluster. 

4. Determine the Size of the Sample . The size of the sample 
depends on the minimum confidence interval and maximum sampling 
error that can be tolerated when making estimates from the sample 
to the general population of Pell Grant recipients. The smaller 
the number of institutions included, the graatar will be the 
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sampling error at a specified confidence interval. The students 
must be distributed cunong enough institutions to minimize the 
error but concentrated at few enough institutions to minimize 
travel costs. A sampling statistician will have to be consulted 
to raalce this determination. 

5. Allocate the Sample among Strata and Clusters . If the 
sample is being stratified, the total number of institutions to 
be sampled roust be divided among the strata- It will also be 
necessary to decide how many institutions will be selected from 
each chosen cluster. Just as concentrating the students among a 
few institutions will increase the sampling error, so will con- 
centrating the institutions among a few clusters. On the other 
hand, the fewer the clusters to be visited, the lower will be the 
overall cost of collecting the data. Again, a sampling statisti- 
cian will have to determine how many clusters to select and how 
many institutions per cluster. 

6. Draw the Sample . The final step is to select the required 
number of clusters at randcm and then to choose the appropriate 
number of institutions at random from each cluster. Depending on 
the stratification used, institutions frc«n several strata may be 
chosen from each sampled cluster. 

Task 1.2; Select Student Sample 

Drawing the student sample will be simpler than drawing the 
institution sample. It is assumed that about most recipients 
will be validated; therefore, it will not be necessary to 
stratify the sampling frame by validation status and over sample 
validated students, as was done in Stage One, in order to have a 
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large enough sample to support accurate inferences about ail 
validated students. However, stratifying by size of award should 
be repeated because it may reduce the standard error but cannot 
increase it, at a small cost. 
Procedures 

1. Determine the Size of the Sample * The size of the sample 
depends on the miniraura confidence interval and maximum sampling 
error that can be tolerated when making estimates frcm the sample 
to the population of Pell Grant recipients, and on the number of 
students drawn per institution (the Cxuster size)- As the tables 
in Chapter 3 indicate, narrow confidence intervals, high signif- 
icance levels, and large cluster sizes (small numbers of insti- 
tutions) all increase the number of students or parents required^ 
The size of the sample must also be limited by the funds avail- 
able for data collection'- Because of the interaction between the 
size of the institution sample (which determines the average 
number of students sampled per institution) and the size of the 
st^udent /parent sample, the two sample sizes should be determined 
together by the same sampling statistician- 

2. Determine the Sampling Frame . The sampling frame should 
include all Pell Grant recipients at sample institutions- A 
definite enrolliuent date, such as October 31 (the reporting date 
for the fall Progress Report), should be specified. This is 
especially important for proprietary institutions, which often 
have an almost continuous flow of new enrollments throughout the 
year . 
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"ompil e the Sampling Frame . Compilation of the sampling 



frame consists of securing a list of all Pell recipients attend- 
ing each sample institution* There are two sources of this list: 



recipients by grant amount and institution only for students 
whose S?uRs have been submitted with their institutions' fall 
progress report. Therefore, this source should be used only for 
institutions for which the number of SARs submitted matches the 
number of recipients claimed on the Progress Report. The use of 
PIMS data has three advantages. First, it obviates the expensive 
and time-consuming process, described below, of getting and pro- 
cessing recipient lists from the institutions. Second, it re- 
duces the response burden on the institutions. Third, it avoids 
warning institutions that they will be visited and thus reduces 
the experimental bias which would result if they took special 
pains to reduce institutional error. 

Institutions which do not file their Progress Reports on 
time or do not submit all their SARs will be asked to supply 
lists of their Pell recipients. A letter of request should be 
secured from the Deputy Assistant Secretary and accompanied by 
specific instructions on what is to be reported, the name/ 
current address and telephone number. Social Security number, and 



grant amount for each recipient. Given the small number of 
letters involved, the letters should be individually typed and 
addressed and individually signed by someone with signature 
authority. Printed "Dear Colleague^* letters can be expected to 
be treated less seriously, forcing delays and more expensive 



PIMS and the institutions themselves • PIMS can provide lists of 
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follow-up efforts, A reminder postcard from the Project Director 
should be mailed to each institution a few days after the letter 
of request? a reminder letter/ more urgent in tone, should be 
sent to each nonresponding institution no more than three weeks 
after the first letter. Depending on the time available, a^other 
letter should be sent two weeks before telephone follow-up 
begins. Institutions which have not responded after two or three 
letters should be called by telephone canvassers who have been 
taught the objectives of the study, exactly what information is 
needed, and how urgently it is needed. All communications with 
sample schools should note that participation in tiie study and 
compliance with the request for recipient lists is not voluntary. 

Long lists from large institutions, at least, will have to 
be prepared for automated sorting and sampling. For the Stage 
One study, the largest lists were sampled systematically, only 
the information on every nth student being keypunched. Although 
this procedure introduces another source of sampling error, it 
may be the only practicable way to handle institutions with tens 
of thousands of recipients. Manual processing may be the 
quickest way to handle the smallest institutions. 

4. Stratify the Sampling Frame . The sampling frame should be 
stratified by size of grant. This can be done most easily by 
ordering eajh institution's list of recipients from largest grant 
to smallest. .^xsts obtained from PIMS should also be sorted by 
grant size- 

5. Draw the Sample . The sample of students will consist of a 
separate sample drawn from the recipient list submitted by each 
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institution. The sampling plan devised by the sampling statis- 
tician will specify how many students are to be drawn from each 
institutvion or how the number drawn is to be calculated for each 
institution* Each institution's list having been ordered by 
grant size, a systematic sample will be equivalent to a sample 
stratified by grant size. If the number of students drawn from 
each institution is the same, the skip interval will be differ- 
ent; in ar»y case, a different starting point should be selected 
at randc^ for each institution. 

TASK 2: INTSRVIEW STUDENTS AND PARENTS 

This task involves visitiig four groups of subjects; depen-- 
dent students, parents of dependent students, independent stu- 
dents, and parents of independent students. However, except for 
the questions asked at the interview, the data collection process 
is the samer therefore, the specifications set forth here will 
deal with all four groups as a whole rather than individually. 
Participation is legally required of all but the parents of inde- 
pendent students, but every effort should be made to obtain maxi- 
mum cooperation from this group* The data collection itself 
includes both an interview and abstraction of data from documents 
furnished by the interviewee* 

Task 2,1: Develop Data Collection Instruments 

A separate instrument must be developed for each of the four 
groups of subjects. Although many of the questions will be asked 
of all four groups, each must be asked a set of unique questions 
because of its status; some of the answers will be used to de -er- 
mine Whether t\e correct status claimed . 
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The instruments used in the Stage One study can serve as a 
basis for instrument development. This will reduce substantially 
the time and effort needed for this task. 
Procedures 

1, Specify Measurements and Measurement Levels 

3aaed on the discussion in Chapter 2, decide what measure- 
ments must be made to perform the desired measurements of program 
error. Specify what level of measurement and verification is 
acceptable (from the student's or parent's word to notarized or 
IRS documentation)- 

2. List Data Needs 

Draw up a list of the specific information items needed to 
carry out the measurements specified in 'step 1, indicating for 
each item whether it is needed from all jfour groups or, if not, 
for which ones . 

3. Collate Stage One Instruments 

Compare the list of data needs with the items included in 
the appropriate Stage One questionnaire to determine which items 

can be dropped from that instrument and w^^ich items must be 

I 

added. 

i 

4, Evaluate the Stage One Instruments 

Interview personnel at Westat, Inc., about the Stage One 
questionnaire, especially the data collection supervisors, coding 
supervisors, and others involved in any debriefing of inter- 
viewers or coders, Ask whether any questions proved to be vague 
or ambiguous, use terms unfamiliar to most respc .dents , cover 
topics which respondents could not answer, or havt any other 
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problems. Ask staff involved in coding, interviewing, and data 
analysis whether the response categories were adequate for ques- 
tions that might be repeated ; whether some of the categories 
provided were used rarely or not at all; whether there were any 
f reouent answers which did not have categories ; and whether some 
open-ended questions could have been provided with codes. 

5 . Write Revised Draft Instruments 

Delete questions which are not needed, add new questions at 
the appropriate points in the instrument, and revise any ques-- 
tions criticized in step 4. To maintain data comparability from 
one error study to the next, major revisions in existing 
questions (for instance, changing frcOT an open-ended to a 
closed-ended question) should be considered very carefully and 
made only for strong reasons • 

6 . Test Instruments 

After the instruments have been revised they must he tested 
with representatives of the groups to v^ich they are addressed ♦ 
Each questionnaire may be administered to a maximum of nine 
respondents without requiring approval by the Federal Educational 
Data Acquisition Council (FEDAC)? this number should be sought 
for field testing ♦ The field test respondents need iK>t comprise 
a statistically representative S2unple of their population; 
rather, they should be chosen deliberately to include the types 
of respondents most liKely to be encountered during real data 
collection • The two student groups (independent and dependent) 
should include married and single students, full-time and part- 
time students, on-campus and off -campus students, dnd students at 
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the five major types of institutions (public four-year, private 
four- year, public two-year, private two-year, and proprietary) . 
The two groups of parents (of independent and dependent students) 
should include parents of the same types of students and, also, 
married, single, and remarried parents r if their cooperation can 
be obtained, the parents of the student field testers can be used 
to test the parent. 

Field tests can be arranged with the cooperation of local 
institutions. To spread the burden of field testing, different 
institutions should be asked to supply testers than were asked in 
Stage One- 

The field tests should be conducted as realistically as pos- 
sible. Ask local institutions to refer students in the various 
categories to you and then call and ask for an appointment. 
Unlike the regular survey, participation in the field test is not 
.Tiandatory. Acl-ninister the questionnairp.'* to the students and 
parents in their homes or dormitory rooms just as if an actual 
survey were being done. Go through the entire questionnaire, 
asking all the questions but noting not only the tester's answers 
but also any difficulties or misunderstandings. At the end of 
the interview, ask a few additional questions about the respon- 
dent's opinions of the questionnaire — which questions were the 
most difficult, whether any were vague or irrelevant, etc. 

The field test is also an opportunity to test some of the 
mechanics of data collection. The time required to administer 
the instrument should be noted, although well-trained field staff 
will probably take less time- 
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7 . Revise Instruments 

Collect the comments, criticisms, axid suggestions of the 
field testers and analyze them for consistent patterns . Make the 
revisions these patterns suggest. When revising a question, be 
sure to make the change on all the forms on v*hich that question 
occurs. If a question causes difficulty for one group of respon- 
dents but not for other groups, some difficulties may have to be 
accepted in order to keep the wording identical for all groups. 
3. Secure FEDAC Approval 

All questionnaires to be administered to 10 or more respon- 
dents must be approved by FEDAC. /^proval requires the comple- 
tion of a standard form (SF-83) and the submission of a support- 
ing statement detailing the respondent burden (the number of 
hours required to answer the questionnaire), the reasons for 
asking each question, the uses of the data, the absence of suit- 
able data which have already been collected, and the cost of data 
collection. Data collection activities must be listed in the 
Federal Register each February for the following year and must be 
included in the agency's "information collection budget." FEDAC 
clearance normally takes several weeks ; .jtw months should be 
allowed in the schedule. FEDAC may suggest or require changes in 
or deletion of individual questions. 

The supporting statement can be based on the statement sub- 
mitted with the Sta^e One clearance application. This should 
reduce both the preparation time and the clearance time, since 
many of the questions will have been justified and cleared 
already. 
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9. Produce Final Instruments 

After any revisions have been made to satisfy FEDAC/ and 
clearance has been obtained, the questionnaires can be printed 
for use in the field. Order at least 25 percent more question- 
naires than expected respondents in order to have enough for 
office and administrative use, training, replacement of losses 
and spoilage, and a field margin caused by transfer of case 
loads, dismissal or resignation of interviewers, etc., which may 
not be accompanied by immediate recovery of the cpiestionnaires 
supplied-. 

Task 2.2t Recruit Interviewers 

To interview students and parents in a national sample, a 
large number of con;petent and well-trained interviewers are 
needed, located throughout the country. The student sample is 
clustered at a limited (if large) number of institutions which 
are clustered geographically themselves. Students can be inter- 
viewed in their dormitories or other housing close tc their 
campuses. In roost cases, their parents will also live nearby, 
but there is no assurance of this for ciny particular parent — some 
parents will live in areas fau: frcwn sample institutions. 
Interviewers must be able to cover these areas as well as the 
locations of sample institutions. 

Given the number of interviewers needed and their disper- 
sion, the most efficient approach would be to contract the 
student and parent interviews to a survey research organization 
which already has a national corps of experienced interviewers on 
call. The procedures specified in thef following for OSFA or some 
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other organization without an established corps of interviewers 
to hire an interview stnff should demonstrate the impractical it y 
of this approach. The entire process can be expected to take 
several months • 
Procedures 

!♦ Determine Interviewer Qualifications 

Student and parent interviewers roust be responsible eind 
reliable^ able no complete their assignments without full-time 
• supervision ♦ They must be intelligent enough to be trained 
quickly and able to respond or adapt to unpredictable situations 
or interviewer responses • They must be articulate in explaining 
the purposes of the survey and able to gain the confidence of 
strangers Who may have good reasons not to cooperate with the 
study^ All interviewers must have moderate arithmetic skills; a 
few truly bi' *ngual interviewers^ who can read# speaks and com- 
r rehend a foreign language (especially Spanish) will be needed ♦ 
They must be able to act in a professional manner and present a 
professional appearance.. In most cases / they will need assured 
access to an autosaobile and a valid driver's license* In many 
areas, they will have to be able to, travel to nearby cities and 
other areas; in sparsely populated aureas^ this may involve dis- 
tances over 100 miles. They also must be able to climb suairs, 
work in adverse weather^ and occasionally conduct several succes- 
sive interviews without a break. If interviewers are to be hired 
as temporary/ part-time Civil Service workers, these requirements 
will have to be converted, into Civil Service qualifications. 
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Although college students often make good interviewers 
---^ecaufle many of them have the requisite intelligence, articu- 

lateness, and flexible hours, care should be taken in hiring thara. 
for this project because they may identify too strongly with the 
respondents and be reluctant to "inform" on "coHc*'^7ues ." No 
student should be hired to interview other students at his own 
institution • 

to 

2. Determine Number of Interviewers Needed 

The number of interviewers needed is a functioh of the num- 
ber of students sampled, the time period available for the inter- 
viewing, the length of the interviews, and the dispersion of the 
interviewees. More interviewers will be needed the larger the 
sample, the shorter the time available for completion of the 
interviews, the longer the interviews, and the more dispersed the 
interviewees. Sample size and dispersion of respondents will 
have been determined before the sample was drawn- The length , of 
each interview can be estimated from the field test,. As 
explained at the beginning o^ this chapter, the field period 
should be as short as possible to avoid introducing a time 
variable into the study. 

For the Stage One study, Westat hired over 200 people to 
complete 8,155 half-hour interviews in 10 weeks. Therefore, the 
average interviewer was able to complete only about four inter- 
views per week. 

3. Determine Distribution of Interviewers 

Plot the addresses of samp led students and their parents by 
three-digit zip code area. In Stage One the average interviewer 
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complet d 40 interviews in 10 weeks, but this rate varied accord- 
ing to the density of interviewees. In sparsely populated areas, 
1 interviewer t'. iO interviewees may be needed to complete half- 
hour interviews at the san» rate; in densely populated areas, 1 
to 80 may be sufficient. Using these or other appropriate 
ratios, distribute the total number of needed interviewers among 
zip areas or groups of adjacent zip areas. 
4. Recruit Applicants 

Given the extensive qualifications for interviewers listed 
above in step 1, it will be necessary to have a large pool of 
applican~-'j for the interviewer positions. fiJoreover, interviewers 
will be needed trom all parts of the 48 states. Some candidates 
may be secured frcMn 0PM files of applicants from other positions, 
and some existing permanent Federal civilian personnel may be 
assigned to temporary duty as interviewers. However, the remain- 
ing specifications for this subtask describe a procedure for 
hiring special temporary Pell Grant interviewers analogous to 
Census interviewers. 

Compile a list of active survey research organizations, then 
write to them for assistance in identifying competent, experi- 
enced interviewers. Describe the purpose of the QC study, the 

f 

role of the student and parent interviews, the interviewer quali- 
fications, and When the interviews arc scheduled to take place. 
This last item is in^xjrtant because roost survey researchers do 
not have surveys in the field at all times everywhere. They do 
not want their interviewers pre-empted by mother study, but they 
want their interviewers to be employed as continuously as 
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possible to keep them from taking other jobs and becoming un- 
available for their own projects. The letter should be assigned 
by as high an official as possible, at least by the Deputy 
Assistant Secretary. S<xm organizations have interviewers all 
over the country, but many cover only specific regions, states, 
or metropolitan areas. 

After a list of recommended interviewers has been compiled 
from the responses to this mailing, send a form letter to the 
people recommended inviting them to apply. Include not only 
background information about the survey, the interviewer require- 
ments, and the schedule, but also business details such as rate 
of pay, and employee, status (whether full-time or part-time, 
temporary employee, or independent contractor, etc.). 

This approach may not produce enough candidates in some 
parts of the country. In those areas, it will be necessary to 
place newspaper advertisements to recruit applicants. Since 
applicants will probably be needed from scattered areas around 
the country, advertisements should probably be placed through a 
commercial advertising bureau. 
5. Screen and Interview Applicants 

Screening and inter\-iewing will have to be done regionally. 
Form letters inviting applications can include someone to contact 
in each regional office; newspaper advertisements should include 
only the nearest regional office. In Stage One, Westat had seven 
regional supervisors for this project. 

The number of applicants interviewed for each area should be 
at least twice the number of interviewers* needed for that area. 
Interviewing by telephone will probably be the only practical 
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method for isolated applicants in areas v>here few interviewers 
are needed, but interviews should be done in person wherever 
feasible, such as in majcr metropolitan areas. 

For each area, rank the applicants interviewed and offer 
positions to the top-ranked people. Press for an immediate deci- 
sion by applicants so that positions can be offered to others if 
the preferred candidates refuse* 
Task 2.3; Train Interviewers 

Although some interviewers may have experience with other 
surveys, that cannot be assumed? all interviewers must be trained 
in basic survey procedures and techniques . They also must be 
taught specific methods for this project ai^ briefed thoroughly 
on the background of :,he study so that they will be able to 
answer respondent questions and will understand the procedures 
they have been trained to follow. It will probably be imprac- 
tical to train all the interviewers in one place at one time- In 
the Stage One study a week was spent training supervisors and a 
week training interviewers. 
Procedures 

1, Develop Training Manuals 

For the Stage One study, Westat developed seven documents 
for training supervisors and interviewers: 

• A two-volume supervisor manual 

• "An Introduction to Interviewing" which included back- 
ground information and a general guide to interview 
techniques, including interviewing ethics and confi- 
dentiality 

• "Home Study Guide for an Introduction to Interviewing," 
consisting of exercises with answers on the material in 
the Introduction 
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• A "Glossary/Docvunant Index" containing definitions of 
technical terras used in the study and samples of the 
forms the interviewers wuld b«* using in the field 

• A -Field Procedures Manual** detailing the prccedur*»« 
for securing appointments and recording and reporting 
results 

• "Question-by-Question Specifications" detailing how to 
record specific answers and how to handle particular 
complicated situations 

A similar set of training manuals will have to be developed for 
this project. Interviewers will take the field procedures man- 
ual, question-by-qu«stion specifications, and glossary with them, 
although they should not need to refer very often to the last of 
these if they have been trained well. Much of the Stage One 
material cam be adapted to this study, especially the question- 
by -quest ion specifications for items repeated from Stage One. 
2. Schedule Training and Notify Interviewers 

One-week training sessions will require paying hotel expen- 
ses for all interviewers, but travel expenses can be held down by 
regionalizing the training, which will also divide the trainees 
into more manageable groups. It is desirable that senior project 
staff be present at each training session to supervise and answer 
difficult questions; this limits the number of sessions which can 
be held tuiraultaneously , There should not be more than a week 
betwecBi the end of interviewer training and field interviewing, 
lest interviewers forget too much of what they learned- I'l Stage 
One, four training sessions were held over a period of two weeks; 
this period could be extended to a third week by allowing inter- 
viewers trained during the first week to begin interviewing while 
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other interviewers were being trained in the third week . Never- 
theless, there should not be more than six training sessions, 
three per week for two weeks or two per week for three weeks . 

Interviewer training schedules si ould be determined early 
enough so that interviewers can be notified of the dates and 
location of their training when they are hired. If that is not 
possible they should be at least told of the date, or they should 
be notified of the training schedule as soon after hiring as 
possible. 

3. Send Training Material to Interviewers 

Send each interviewer a copy of the introduction to inter- 
viewing, the home stjdy guide, and the glossary, with instruc- 
tions to study these items before the training session. Pay the 
interviewers for a reasonable amount of time to spend studying, 
notify them that there will be a quiz on the material early in 
the training, and administer a quiz after the training orienta- 
tion. 

4. Train Field Supervisors 

The regional interviewing supervisors need to be even better 
informed about the purposes of the survey and the field tech- 
niques than the interviewers, since they will be responsible for 
initial decisions when interviewers do not know what to do. They 
also must be thoroughly trained in the supervisory methods which 
will be used to assure the quality of the interviews. Since they 
will have to be experienced in both survey management and field 
interviewing, they will not need as much training in general 
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survey mothods as the interviewers • Tne time saved can than be 
spent on supervisory procedures. 

The supervisor training should include the following topics 

• The background arid purposes of the project 

• Item-by-item specifications for the questionnaires 

• Project: field organization 

• Quality control procedures for verifying interviews 

• Procedures for reporting to the central office and 
forwarding questionnaires 

About five days should be spent to cover these topics thoroughly 

5. Train Interviewers 

Interviewer training should use a variety of methods for 

efficiency and to hold the interviewers' interest. These may 

include : 

• Formal lectures, illustrated where appropriate 

• Training films 

• Demonstration interviews 

• Practice interviews between individual trainees and 
instructors 

• Practice interviews between pairs of trainees 

• Quest ion-and-answer sessions 

The progress of the interviewer trainees should be monitored 
V 'h formally (through graded quizzes) and informally (through 
ob'jervation by the instructors) throughout? trainees who cannot 
m€<et minimum standards must be permitted into the field. 

Lectures . Lectures are appropriate for the preaentaf.on of 
wholly new material, such as e;cplaining the background and pur- 
poses of the study, introducing general interviewing techniques, 
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and describing administrative procedures* Trainees should be 
encouraged to ask questions during the lectures, and experienced 
interviewers can be called on to give examples of interview 
methods. Overhead slides or other illustrations such as handouts 
should be used to outline the material amd emphasize essential 
points . 

Training Films . Films or filmstrips are helpful in pro- 
viding variety in the trainii^ sessions and in capturing the 
trainees' interest. Films on general interview techniques can be 
used for this purpose. 

Demonstration Interviews . After the purposes of the study 
and general intei-view techniques have been introc uced, a demon- 
stration interview, in Which one instructor plays the student or 
parent and another the interviewer, can lend a greater sense of 
reality to the training. The demonstration interview may precede 
or follow an item-by-item lecture on the questionnaire. At least 
two demonstrations should "oe included: one of a parent interview 
and one of a student interview. A good practice during. a demon- 
stration intervi«w is to have a third instructor coding a ques- 
tionnaire in response to the demonstraticm answers, with the 
trainees roarXing copies of their own and checking their codes 
against the overhead. 

Practice Iirterviews . It is important that each trainee 
practice each of the interviews at least once; skip patterns and 
other difficult sections should be practiced more often. Initial 
practice interviews should resemble the demonstrations, with an 
instructor taking the role of the parent or student and other 
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trainaes obsarving. Instructors should tak« careful note of any 
mistakes made by the trainee interviewer but should not interrupt 
the interview except to correct a major error such as following 
the wrong skip pattern. Trainee observers can be asked to list 
mistakes at the end of the demonstration; for diacussion by the 
group, those not listed should be brought up by the instructor. 
Trainees ^o are experienced interviewers should participate in 
these demonstrations. 

Given the amount of time and the number of instructors prob- 
ably available, trainees will eventually have to be paired, -one 
doing the interview and the other playing respondent. If these 
practice interviews are done in various corners of a large room, 
an instructor can be available to answer specific questions on 
signal (such as When neither member of the pair knows how to 
proceed after a certain point) and to oversee a portion of each 
practice interview. After each round of practices the group can 
be reassembled for a discussion of problems encountered. 

In all practice interviews, the "respondent" should have a 

1 

script or list of answers to the questions, so that the tiraining 
staff can be sure that all trainees encounter the same situation 
and cover the same material. 

Question-and-Answer Session . Open opportunities for all 
trainees to ask questions should be built into the training 
schedule. At least one should be devoted to each questionnaire 
and one to field procedures. A summary session in which ques- 
tions can be asked on anything about which interviewers are 
uncertain should be scheduled near the end of the training. 
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Evaluation and Retraining * Minimura training standards 
should b« astablished before the training begins. Formal quizzes 
and observation should be used to evaluate the trainees . Those 
who appear to be having trouble learning important items, or who 
do not get minimum scores on the quizzor roust be given addi- 
tional training in the evening, after the regular schedule. 
Trainees who still cannot meet the training standards must not be 
allowed into the field. 
•ask 2.4t Contact Students and Parents 

An unexpected telephone call from a person whom the person 
being called does not Tcnow is an unfavorable context in Which to 
secure an appointment for a survey interview, e&pecially when the 
survey is of the ncture of an investigation. Therefore, sampled 
students and parents should be informed of the survey before hand 
in a less threatening context. 
Procedures 

1. Draft Letters and Forms 

Write letters for the signature of the director of the Divi- 
sion of Quality Assurance or of the Deputy Assistant Secretary, 
informing independent and dependent students of thier selection 
for the survey, and informing parents of independent and depen- 
dent students of the selection of their children for the survey 
(four somewhat different letters in all). Include in the letters 
information about the purpose of the study, a notification that 
an interviewer will be calling to arrange an appointment, 
instructions on what documents to have ready for the interview, a 



4-31 104 



reminder that participation is not voluntary but part of tha 
terms of tha Pell Grants and a telephone number to call for 
further information. 

Compile a checklist of all the documents which might be 
requested from any of the four types of respondents. The fol- 
lowing should be included in the letters 

m An ''IRS Form 4506, Request for Copy of Tax Form** if the 
SAR indicated the student^/parent filed a tax return 
with a request that they sign it, provide certain Icey 
information such as the parents Social Security num- 
bers, and return it to the project office 

• A request to respondents who indicated that they own a 
home to provide the name and address of their tax 
assessor's office 

• A release form to those who reported APDC benefits with 
a request that they provide the name and address of 
their local public assistance office 

• Several copies of a "Financial Institution Authoriza- 
tion to Release Information** to those who claimed to 
have more than $4,000 in checking and savings accounts 
at the time of application 

2* Compile Mailing List 

LxSt the mailing address of each student in the sarople. A 
current address for each Pell recipient should have been included 
on the recipient lists from which the student sample was drawn • 
Parent addresses should also have been included but may not have 
been, especially for parents of independent students • Write 
again to institutions which cxnitted the addresses, enclosing a 
list of the students sampled and requesting current student and 
parent addresses # 

For students drawn from PIMS files because their schools had 
sent in the SARs of ail current recipients with the fall Progress 
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Reports, student but not par«nt addresses should also toe avail- 
able. Write to these students' institutions for missing student 
addresses and for parent addresses. 
3. Send letters and Forms 

Secure enough envelopes with the project address and 
"Address Correction Requested" printed on them (allow about 10 
percent extra for spoilage r replacements, etc.)' The Postal Ser- 
"ice is supposed to provide forwarding addresses for addressees 
who have moved for 25 cents per Eiddress. To the extent that this 
is done, tracing students and parents will be easier. 

Send the appropriate letter to each student and parent, 
enclosing the check list indicating the documents required from 
each individual and the tax return release form. 
Task 2.5t Conduct Int.erviews 

Parents and students should be assigned to individual inter- 
viewers by regional interview supervisors on the basis of geo- 
graphic propinquity, interviewer preference, and language. 
Interviewers will have to set their own schedules. 
Procedures 

1. Assign Students and Patents to Interviewers 

The number of interviewers hired in each region and zip code 
area or group of areas should correspond to the estimated number 
of students and parents to be interviewed there. In some areas, 
there will be just one interviewer, to whom all the respondents 
will have to be assigned. Large metropolitan areas and, perhaps, 
few communities where large public universities are located, will 
hav^ several interviewers. Regional interview supervisors should 



assign about hal£ o£ their intarviaws in thasa areas. In most 
cases eadh subject should be assigned to the nearest interviewer, 
based on the respective addresses. If the supervisor has reason 
to suspect there may be a parental language problem, and a bi- 
lingual interviewer is available, the bilingual interviewer 
should be assigned to those parents. In many cases, it will be 
possible for a monolingual interviewer to conduct the interview 
through an interpreter, such as a family member who speaks 
English. All students may be presumed to speak English. 

Set aside time in the training schedule for interviewers 
from metropolitan areas to trade assignments based on the n'eigh- 
borhoods they would prefer to visit. Regional supervisors must 
approve these arrangements to hold down travel costs. The 
training session will probably be the only time the interviewers 
are together . 

Attempts by interviewers to schedule interviews will reveal 
some address changes. In most cases, the subjects will have 
moved only a short distance, and the interviewers can schedule 
interviews as if no move had been made. Long-distance moves will 
require referral to the cases to the regional supervisor for 
reassignment to another interviewer ^if the respondent stayed in 
the S2une region) or, through the regional supervisor, to the 
supervisor in another region. 

A« signments for these reasons are made, the overall 

load o. . interviewer will become more apparent. After about 
a third oZ the field period has passed, regional supervisors 
should assign the remaining cases- 
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2. Schedule Interviews 

Instruct interviewers to attempt to schedule individual 
intervievrs in advance by telephone Whenev^^r possible ^ and provide 
thean with scripts for doing so. The first, fev days of the field 
period should be spent making these telephone contacts. 

Some respondents will not have telephones/ or attempts to 
reach them by telephone vill be unsuccessful. Instruct inter- 
viewers to substitute personal visits in these cases. They 
shoL'id conduct the interviews immediately Whenever the respon- 
dents will permit it. 

3 . Deal wi th Refusals and Avoiders 

Interviewers should remind refusers and avoiders (subjects 
who do not refuse but consistently fail to keep appointments or 
find excuses for not making them) that participation in the study 
is required (except of parents of independent students) only 
after trying to gain cooperation voluntarily by answering ques- 
tions (explicit or implied) and attempting to conform to the 
subjects' schedules. 

4. Conduct Interviews 

Interviews for the Stage One study were designed to average 
about half an hour each — half that for parents of dependent stu- 
dents. Therefore, only a minority of interviewers' time was 
spent actually conducting the interviews. A half hour is usually 
considered to be about the maximum time for which a respondent's 
attention can be held without causing impatience, hostility, or 
an outright break -off. Therefore, questions should not have been 
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added to the Stage One questionnaire unless oth«r questions which 
would take about the same time were deleted. 

Note that the "interview" includes not only questions asked 
by the interview and answered by the respondent, but also the 
interviewer's examination, on the spot, of documentation of the 
recipient's income and status. 

5. Edit Questionnaires 

Interviewers must edit their questionnaires as soon as pos- 
sible after the interview — no later than the end of the same day. 
Nonstandard abbreviations used in recording comments or open- 
ended answers roust be explained, illegible handwriting must be 
clarified, and zeroes must be filled in where appropriate. All 
answers must be checked against the question-by-question specifi- 
cations and errors, omissions, or anomalies attributed to the 
respondent or interviewer. 

6. Return Questionnaires 

Establish field procedures for the shipment of survey in- 
struments to the project office for coding and data entry. They 
should be routed through regional supervisors for quality control 
(see next subtask) and sent in at least once a week. 
Task 2.6: Implement Quality Control an d Supervision Plan 

Quality control measures in the field are specified here. 
Two other quality control measures are specified elsewhere: 
immediate post- interview editing by the interviewers in Subtask 
2.5 and editing at the project office in Chapter 5. 
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The steps described here are vital in assuring that the data 
will be as complete and correct as possible # collected on time, 
and within budget. 
Procedures 

1 . Coordinate Interviewers 

Reassign cases which are potentially completions (i.e., are 
not important to ccxnplete because of untraceable address, refu- 
sal, death of respondent/ etc) but cannot toe completed by the 
interviewers to whom they ware first assigned. Assign new cases 
Which have been added to the region because of respondent moves 
and cases which were held back at the beginning of the field 
period . 

2. Edit Questionnaires 

Edit a sample of questionnaires transmitted by interviewers. 
A few cases chosen randomly should be edited item-by-item to make 
sure that interviewers understaxid how to fill out the question- 
naire and are doing proper field editing. The first few cases by 
each interviewer should be edited in this way (the first three by 
each interviewer were done in Stage One). Report interviewer and 
repondent errors to the project office for the tabulation. If a 
systematic error appears across a number of interviewers, there 
may be a problem with the instrument? a solution will have to be 
devised and all supervisors and interviewers informed of the new 
procedure . 

3 . Verify Interviews 

Verify a small random sample of each interviewer's work by 
telephoning the respondent and confirming the date and time of 
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the intarview, asking whether the interviewer asked to see key 
documents and actually reviewed them, reasking a few key ques- 
txons, and inquiring Whether the interviewer was impolite, hos- 
tile, or late for the interview. Rei/.terviewing is the only way 
to determine What interviewers are really doing in the field and 
the only way to guard against -armchair interviews," the filling 
out of instruments with bogus data without ever visi'-ing the 
respondent . 

If an armchair interview is discovered, all of that inter- 
viewer's completed cases must be verified. The interviewer must 
be dismissed and uncompleted or bogus cases reassigned. Failure 
to review documents or ask key questions may indicate interviewer 
dishonesty or merely a misunderstanding of the questionnaire. 
The interviewer should be called immediately and the situation 
clarified; if he is seeking shortcuts or deliberately evading his 
responsibilities, he must be dismissed. Less serious problems 
such as rudeness or an occasional missed question must be 
resolved at the next regular telephone conference (see step 4, 
below) . 

4. Supervise Interviewers 

Keep careful records of the assignment of cases to inter- 
viewers. Schedule a regular weekly time for each interviewer to 
report to his supervisor by telephone the status of each case 
assigned to him (interview completed and questionnaire forwarded 
for coding, refusal, still trying to contact respondent, inter- 
view scheduled, etc.). Discuss any of the less serious problems 
mentioned above which have been found in verification. Skipping 
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questions and int«rvi«wer ruden«»» Can comprise the study if they 
are widespread and consistent; they do not require immediate dis- 
missal of the interviewer, but they must be dealt with firmly. 

If an interviewer does not have good reasons for not com- 
pleting interviews at the expected rate, some of his cases may 
have to be reassigned. Some interviewers will have runs of bad 
lucX during vAiich respondents will be tinlocatable, evasive, and 
uncooperative; cars will breakdown; weather will interfere with 
travel; and they will fall ill. When these problems are cited 
frequently over several weeks, the supervisor must try to deter- 
mine Whether the interviewer *.a really fulfilling his responsi- 
bii.itief,. 

5. Assign New Cases 

Refer respondents who have moved to different regions to the 
supervisors of those regions; the supervisors will be responsible 
for assigning those cases to individual interviewers. As some 
interviewers complete their initial assignments, the supervisor 
will also be responsible for assigning the cases Which were held 
back initially. 

6. Control Expenses 

Require interviewers to report to their supervisors weekly 
yyy telephone on the time they spend in the field and their expen- 
ses. Supervisors should forward these reports to the project 
office. Where checks should be issued promptly; this will greatly 
enhance interviewer morale. Telephone reports should be followed 
by signed statements and receipts sent to the field supervisors, 
who should examine them before sending thera on to the project 



ERIC 



4-39 



112 



office. Supervisors should compare each interviewer's time 
report with the activity (interviews completed, appointments 
made, appointments broken by respondents, etc.) reported for the 
same week, checking to see that acccMopIishments are reasonable in 
relation to time charged and that expenses such as mileage cor-r 
respond to the type of area and nun*. ->er of appointments. Appar- 
ently unreasonable charges must be discussed wit;h the inteviewer 
during the next telephone report. Amounts disallowed by the 
supervisor should be deducted from the interviewer's next pay- 
ment. Final payments should be withheld from interviewers who 
have had more than one such disallowance until all accounts are 
reconciled. 

Give the interviewers an explicit statement of what expenses 
. will be allowed when expense reporting is discussed during train- 
ing. If expenses are allowed for meals, place a reasonable limit 
on the amount. Expenses should be approved by supervisors in 
advance for any trips to isolated su'ijects which require over- 
night travel and acconanodations. 
Task 2.7; Follow up by Telephone 

When missing, illegible, illogical, or inconsistent data are 
discovered in any of the data preparation steps — receipt, manual 
editing, or machine editing — an effort roust be made co contact 
the responsible interviewer, the student, or the parents. Based 
on past experience, this task, although extremely important in 
ensuring complete emd reliable data, does not require a large 
amount of staff time. In the Stage One study, Westat recontactad 
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only 340 students and parents out of a total sample of roughly 
8,000. 

Several of the coding staff should be trained to conduct the 

telephone intervievfs* Training need not last longer than one 

day* It should cover the following topics; 

e Beginning the interview — ^how to gain cooperation and 

establish rapport with a respondent during the intro- 
duction 

• Using the guestionnaire ~how to ask the questions 

• Probing — how and when to probe the respond er for addi- 
tional information 

• Editing the interview — how to end the interview with 
the respondent 

The training session should end with practice telephone calls 
that simulate different types of respondents and responses. 
Procedures 

1, Send Instruments to Telephone Station 

Coders and receipt clerks refer all cases that have omis- 
sions, illegible answers, or illogical responses to a coding 
supervisor \^o decides Which cases warrant telephone follow-up « 
On those cases that do, the supervisor decides who should be 
called, the field interviewer or respondent, then forwards the 
instruments to the telephone interviewers « 
2* Telephone Respondent or Interviewer 

The telephone staff calls students, parents, ami inter- 
viewers as cases are referred. Telephone numbers for students 
and parents should be available from the respo:.dent control file 
created during the s^^Ilpling stage of the study* The telephone 



114 

4-41 



staff must b« diligent in attempting to contact and get the coop- 
eration of students and parents* 
3 . Code Response 

Specific items are receded based on the new data collected 
in the telephone conversation. 

TASK 3 1 COLLECT SECOtfDARY DATA 

The primary objective of the student and parent data collec- 
tion effort is to obtain data that validate information on the 
Pell Grant application. During the field work, interviewers asX 
siudents and parents to show them verifying documents. Specific 
line item dollar amounts from the documents are then coded on the 
interview forms. There are several problems with collecting 
documentation only from students and parents during the in-person 
interview. Many respondents, even with considerable advance 
notification, will be unable to provide documents at the time of 
the interview. In other cases, the documents will not validate 
the information entered on the application because they are not 
up to date as of the time the application was submitted. 
Finally, much documentation received directly from students and 
parents will not be totally reliable- For example, students and 
parents will often show the interviewer uncertified working 
copies of their tax returns. 

In light of these documentation problems, it is strongly 
recommended that additional major validating documents be 
obtained directly from the issuing institution or governmental 
agency. In the Stage One study, documents were collected from 
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the IRS to verify adjusted gross income, taxes paid, and medical 
and dental expenses, from banks to verify cash and savings, and 
' from local tax assessors to verify home value. For future error 
assessments, documents could also be collected that would vali- 
date AFDC now that it is a separate line item on the application 
form. (Prior to 1981-82 all nontaxable income was included in 
one line item.) In addition, computer tapes could be collected 
from the IRS as additional v^erification. 
Task 3.1; Collect Hard Copy Secondary Data 

Procedures for collecting hard copy verifying documentation 
are relatively straightforward. Student Aid Reports (SARs) are 
collected from each sampled institution and then reviewed by a 
home office coder, with the guidance of the professional staff, 
to determine which documents will be needed to validate the 
information entered on the application. The respondent is then 
provided with a letter for each institution or agency from which 
documentation is needed. The letter, which must be signed by the 
respondent, will authorize release of the information directly to 
the project officer. Once the authorizing letters are returned 
to the project office, they are sent to the appropriate agency. 
Experience suggests that roost agencies need five to seven weeks 
to provide the requested data and that the entire process — from 
r€M[uesting SARs from institutions to receiving documentation from 
the appropriate organizations or agencies^ — requires 16 to 20 
weeks . 



116 

4-43 



Procedures 

!♦ Request SARs from Instttutiong 

Send a list of the students selected for study participation 
to each sampled institution with a request for copies of the SAR 
on file for each of the selected students and a current mailing 
address and phone number. 

2. Telephone Institutions That Do Mot Respond 

Tv#o weeks after mailing the initial letter, begin to call 
institutions that have not sent SARs. Of the 307 institutions 
sampled for participation in the Stage One study, 305 institu- 
tions cooperated in sending copies of the SARs. 

3. Review and Key Enter SAR Data 

. Using procedures detailed in Chapter 4, "Data Preparation," 
code addresses and SAR information onto a coding sheet. From a 
review of the SARs, list the respondents needing a release form 
fcr the IRS, AFDC, banks, and/or tax assessors. Key punch data 
from the SAR coding sheet. The SAR file can be used to create 
mailing labels, instrument labels, and a Master Receipt Control 
Log (see Chapter 4). Eventually the SAR file must be merged with 
other data sets collected during the course of the study. 

4. Send Release Forms and Other Information to Students and 
Parents 

Mail students and parents a letter of introduction and a 
package of materials to assist them in preparing for the Inter- 
view. Include in each package a list of documents that the stu- 
dent or parent will be asked to show the interviewer. "Custom- 
ize" this list for each respondent, base»d on information from the 
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SAR. The following should be included in the package i 



An "IRS Form 4506, Request for Copy of Tax Form" if the 
SAR indicated the student/parent filed a tax return 
with a request that they sign it, provide certain key 
informaticm such as the Social Security numbers of both 
parents (if dependent) or the Social Security number 
of the spouse (if independent), and return it to the 
project office 

A request to respondents Who indicated that they own a 
h<»ne to provide the name and address of their tax 
^assessor's office 

release form to those Who reported AFDC benefits with 
a\r€H}uest that they provide the name and address of 
their local public assistance office. 

Several copies of a "Financial Institution Authoriza- 
tion\ to Release Inf oxmanion" to thoae who claimed to 
haveAroore than $4,000 in checking and savings accounts 
at the time of application 

Finally, it is important that an "information update sheet" be 

enclosed in the package for the student or parent to fill out 

with current names, addresses, and telephone numbers. To ensure 

a prompt response, a preaddressed postpaid envelope labeled with 

the respondent's study identification number should be enclosed 

in the package. (Sec Task 2.4 for further details.) 

5. Telephone Students and Parents That Do Not Respond 

To ensure that the project's schedule does not lag, tele- 
phone students and parents who do not return their release forms 
within three vfeeXa* 

6. Obtain IRS Forms 

Once all IRS Request for Copy Forms are returned, send them 
to the appropriate regional iRS Service Centers. Expect five to 
seven weeks for the IRS to return photocopies of tax returns, 
longer if the request is made between January and April, the 
busiest time of year for the IRS. 
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7. Obtain Tax Assessor Records 

Verify tiw names and addresses of local tax flissessors pro- 
vided by students and parents against a roster of names and 
addresses that U available from the International Association of 
Assessing Offices, Once the addresses are verified, send letters 
to the assessors Which provide the name and address of each home 
owner and Which request the follo*d.ng information: 

• Frequency of assessment 

• Assessment value as of date of Pell Grant application 

• Formula for determining the fair market value 

It may be necessary to contact some of the assessor offices by 
telephone in order to clarify the request and obtain the infor- 
mation. 

8. Obtain Documentation from Financial Institutions 

Once they are returned from students and parents, send 
release forms to banks and other fxnancial institutions. 

9. Obtain Documentation from Public Assistan ce Offices 
once they are returned from students and parents, send 

release forms to public assistance offices. 
Task 3.2: conduct IRS Tape Match 

It may be desirable to conduct an identifiable tape match 
with IRS data in addition to— or in lieu of —collecting hard copy 
1040 and 1040A tax returns from regional IRS Service Centers 
(Task 3.1, .Step 6). In deciding whether or not to conduct a tape 
match, the following issues should be considered: 

• Timing of the Availability of IRS Data 

The tax return filing d eaAline is April 15. However, 
full tax return data are often not available on 



ERIC 



119 



4-46 



comput-or tap« unt.ll October or Novarober of th« same 
calendar year. This fact should not advarsely affact 
tha schedule of the error study If, as recommended 
earlier In this chapter* the recipient sample Is not 
dra%m until late September or 9arly October . 

Logistics 

The IRS requires that all d ; .atches occur at one of 
its facilities. This adds logistical complications and 
may add to the tisie required to complete the error 
study. 

Legality of Data Hatch 

Currently f le^jislaticm and regulations governing the 
IRS require that a release form Which meets IRS 
requirements be signed by aach tax payer before 
individual IRS data cem be released. Release forms 
would have to be collected from recipients and parents 
(TasX 3.1) if either a tape match is conducted or 
hard copy tax return data are collected and processed. 

Match Rate 

A major shortcoming of many tape matches is th^ low 
rate of successful ntatches. Two factors contribute to 
the low rates. First, many Pell Grant recipients do 
not file a tax return. In the approach specified here, 
the SARs of all sampled recipients would first be re- 
viewed; only the release forms and information of those 
who indicated on their Pell Grant application that they 
filed a tttx return would be forwarded to the IRS for 
the match- Second, incorrect linking identifiers are 
used in attempting to conduct the tape match. For an 
IRS tape match, the parents' Social Security numbers 
(SSMs) for dependent recipients and the recipient's and 
spouse's SSNs for independent recipients would be the 
linkage media. As described in step 4, the required 
SSNs must be collected from recipients and parents 
along with signed release forms. ^ Steps should be taken 
to ensure that the SSNs provided the IRS are correct. 
Coding and key punching of 8S^ should be 100 percent 
verified before a tape is sent to the IRS* It may be 
desirable to verify SSNs during field interviews of 
students and parents. 

Data CoH^rability 

Another major concern in tape matches is the equiva- 
lence of the data to be matched. The definitions of 
the data required on the Pell Grant application are not 
al%mys consistent with the data maintained by other 
agencies. For example, data from the Pell application 
on annual VA educational benefits is not compatible 
with data maintained by the VA on VA educational bene- 
fits. However, for roost cases, there should be no 
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compatablllty probl«nis with tha IRS* De£lnitlons for 
AO I and Pdd«ral taxes paid, the tw items of most con- 
cern in a Pell/ZRS match, are identical. Problems, 
ho%irever, arise in cases where parentis of dependent 
recipients are divorced or separated. In these cases / 
the Pell application requires the AG! and taxes of the 
parent %^o supplied more than half of the 4ependent 
student's support. If the parent, remarries and files a 
joint return with the new spouse, or remains unmarried 
but still files a joint return with the former spouse, 
it is impossible to separate the Pell-defined AOI and 
taxes from IRS files. In an error study, the divorced 
and separated cases would not be screened out before 
the IRS file match. Instead, they would have to be 
dropped during the final file merge Which creates the 
"best value" data file for analysis. (See Chapter 5, 
Task 7, for dettails.) 

Procedures 

1. Create Tape to Submit to IRS 

Once all SARs of sampled recipients have been received from 
institutions (Task 3.1, step 1), and release forms and required 
SSNs have been received from students and parents (Task 3.1, step 
4), create a tape to provide the IRS for the record match. The 
special tape should include only the data of those recipients 
whose SAR indicates they or their parents filed a tax return. 
Each record on the tape must include the parent(s) SSN(8) (if 
dependent) or the recipient's and spouse's SSNs (if independent), 
and the stu-ly identifier for each case. 

2. Submit Tape and Release Forms to the IRS 

Provide the IRS with the tape containing the selected 
records and the appropriate release forms. The IRS must match 
the data with their files based upon the parent's SSN (for depen- 
dents) or the student's SSNf (for independents). If a match is 
found, the IRS must add the parent's (if dependent) or student's 
(if independent) AGI and taxes to the record. If no match is 
found, IRS must add a "no match" flag to the record. 
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3. Merge Returned IRS Tape with Study Data 

After the tape ia returned frcJin the IRS/ merge it with other 
study data. (This procedure is discussed in detail in Chapter 5, 
Task 7.) 

TASK 4; COLLECT DATA FROM INSTITUTIONS 

TasX 4»I: Develop Data Collection Instrxaroents 

Three instruments vce developed during tnis task: the Stu- 
dent Record Abstract (SRA) for recording data from student 
financial aid files* the Institutional Questionnaire (IQ) for 
recording the responses of financial aid administrators during 
formal intervievfs, and the Corrections Control Group (CCG) form 
for recording the corrections behavior of a special control 
sample of recipients. 

Instrument design/ although extremely critical to the data 
collection, need not be time consuming since the instruments used 
for each annual error assessment can be developed from the prior 
year's instruments. The exact content and design of the instru- 
ments for the next assessment — and for each subsequent annual 
assessment — will depend both on the data needed to satisfy the 
analytic needs of the assessment and cm an evaluation of the 
prior year's instruments. 
Procedures 

• 1. Design First Draft of Instruments 

Construct a draft of the three inatruicents based on a review 
of the data needs and an evaluation of the effectiveness of the 
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prior year's Instruments. Attention should be paid to the 
fol lowing: 

• Length of Inatriment * The length of the tnatrument is 
very important* \ long Instrument may unnecessarily 
Increase the length of the intervievr, the arooant of 
time spent at each institution, and the %iforkload of 
field representatives I coders, keypunchers, and data 
analysts . 

• Clarity of Questions and Instructions . Interview ques- 
tiona should be «K>rded in such a vray that the financial 
aid officer answers without needing instruction on how 
to respond <^ an explanation of the intent of the 
question* 

• Order of Questions . Interview questions should be 
arranged in a logiczd. sequence. 

• Format . The instrument format should facilitate the 
work of the field representative, coder, keypuncher, 
and systems analyst. 

2. D evelop Other Data Collection Materials 

All materials and forms to vft^ich financial aid administra- 
tors and other Institc.tional officials are exposed must be 
included in the FEDAC/OMB package and, therefore, roust be 
developed at this stage of the study. These materials include: 

• A letter sent to the presidents and financial aid offi- 
cers of the sample institutions describing the study 
and asking f Jr their participation. 

• A letter sent to the financial aid officers outlining 
in detail the procedures the field representatives will 
follow during the site visits, requesting that the 
institution send a list of all Its Pell Grant r'eclp- 
lents for the purpose of drawing a student sample , and 
requesting that the institution give an Indication of 
the time or times most convenient to schedule a site 
visit. 

• A follow-up postcard sent as a reminder to financial 
aid officers who fail to return their list of Pell 
Grant recipients . 

• A record of disclosure, satisfying the conditions o£ 
the Privacy Regulations, Issued to each student whose 
file is reviewed. 
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Further discussion of the purpose and content of these materials 
is found in Task 1.2, "Select Student and Parent Sample" and Task 
4.2, "Contact Institutions and Schedule Site Visits." 

3. Conta<;t Institutions for Field Test 

Contact at least three institutions local to the project 
office and ask them to participate in the field test of the draft 
set of instrtunents . Public, private nonprofit, and private 
institutions should be represent*^! in the field test saiople . 

4. Conduct Field Test 

Visit the participating institutions in two-meiober teeuns and 
enact a typical day of data collection. First, interview the 
financial aid administrator. While one member asks the ques- 
tions, the .jther takes detailed notes on the length of the inter- 
view, the flow of the interview, and any difficulties the respon- 
dent is having with the questionnaire. After the interview, 
discuss the questionnaire with the financial aid officer high- 
lighting any questions that may need improvement. Next, select 
ac random the files of approximately 10 Pell Grant recipients . 
While one member reviews the files, the second takes detailed 
notes on the use of the Student Record Abstract and on any prob- 
lems that might be encountered in locating and interpreting 
financial aid data. 

5. Revise Data Collection Instrvunents 

Analyze the results of the field test and revise the instru- 
ments as necessary. 

6. Conduct Second field Test 

It is advisable to conduct a second field test to appraise 
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the effectiveness of the revise4 instruments. The second test 
need not be as extensive as the first. For the second teat, 
visit two institutions not visited for the first test. Select no 
more than three files to review at each institution. 

7. Revise Data Collection instruments 

Review the results of the second field test and construct a 
final set of instruments. ^ 

8. Prepare and Submit Clearance PacKage 

Once the instruments are final ired, submit the FEDAC/OMB 
clearance package. Preparation of the clearance package need not 
be a time-consuming task since large sections of Advanced Tech- 
nology's Stage One package can be used as "boilerplate" for 
future packages. (Clearance requirements do not change substan- 
tially from year to year.) 

In ccmipleting the package, adhere closely to the instruc- 
tions set forth in the Standard Form 83, "Request for OMB 
Review." The forms clearance package should consist of the 
following: 

A completed SF-83 

Copies of all instruments 

Copies of other data collection materials 

A matrix which links specific research objectives with 
specific items on each instrument 

A supporting statement Which includes the following; 
the study's objectives 

a review of prior related studies and a discussion 
of the circumstances that make this survey 
necessary 
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the analysis strategy 

the types of information to be collected 

the sampling plan and the statistician Who 
approved the plan 

expected response rate and how nonresponse will be 
handled 

estimates of respondent burden and steps to be 
taken to minimize respondent burden 

- pretest results 

discussion of any sensitive questions 

- procedures used to protect confidentiality 
tabulation and publication plans 

a schedule for data collection and publication 

a list of consultations held outside the project 
office 

an estimate of costs to the Federal Government 
Task 4.2i Schedule Site Visits 

The purpose of this task is to: 

• Contact the sampled institutions and request that they 
participate in the study. 

• Request that the institutions send a list of their Pell 
Grant recipients to the home office for the purpose of 
drawing a recipient sample* 

• Schedule the site visits with an eye towards minimizing 
travel costs- 

Procedures 

1. Send Initial Contact Letter to Institutions 

As soon as an institutional sample has been dra%m, send a 
letter to the presidents and financial aid officers at the insti- 
tutions describing the purpose of the site visits, emphasizing 
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their importance, outlining relevant privacy regulations, and 
asking for their participation. 

2. Send Second Letter to Financial Aid Officere 

Send a follow-up letter to the financial aid officers only. 
This letter should include the follo%iring: 

• A request that the institution send a Complete and • 
current list of Pell Grant recipients (see TasK 1.2 for 
details). 

• A detailed description of the procedures the field rep- 
resentatives will follow during the site visit. 

• A request that the financial aid administrators notify 
the registrar, bursar, and other officials at the 
institution from whom information will be collected 
regarding the proposed visit. 

• A request that the financial aid officer inform the 
home office of the time or times most convenient to 
schedule a visit. 

3. Recontact Institutions That Fail to Submit Recipient List 

A series of procedures must be followed to collect recipient 
lists from institutions which do not promptly respond to the 
second contact letter. These procedures include a reminder post- 
card, telephone follow-up, and visit to institutions to draw a 
sample on-site. These procedures are discussed in more detail in 
Task 1.2, -Select Student Sample.** 

4. Construct Initial Master Site Visit Schedule 

Establish a site visit schedule for each field representa- 
tive prior to the training week and insist that each adhere to 
it. By putting time and effort into planning the site visit 
logistics— including planning and making airline, hotel, and car 
rental arrangements— substantial cost savings can be realized. 
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The following is the list of steps to be followed in constructing 
the initial schedule: 

1* Plot institutional sites on large wall map. 

2. Divide map into regions, with each regicm assigned to a 
field representative. Each region should represent 
approximately the same number of work d&ya, travel 
days, and rest days. In an effort to minimize costs, 
every effort should be made to capture geographic 
clusters of institutions vihen drawing these regions. 

3. For each region, draw efficient ^avel cycles. At this 
point it is necessary to investigate airline and rental 
car schedules and rates. 

4* Assign tentative appointment dates to each institu- 
tional site. 

5. Telephone Institutions to Arrange Visits 

After the tentative scbeduling is ccxnplete, call the insti- 
tutions to set appointments, learn of vacaticxi periods, and 
determine the accessibility of their financial aid files and 
other data needed for the record abstracts . 

6, Adjust Site Visit Schedule as Necessary 

Adjust the schedule. Scheduling appointments by phone 
rather than by letter allows for imme<fiiate revision of schedules 
within a region when it: is learned that a tentative appointment 
date is inconvenient for a particular site. 
Task 4.3: Recruit Field Representatives 

To a great extent, the success of the institutional da^a 
collection depends on recruiting individuals who have prior 
institutional financial aid experience. Financial aid profes- 
sionals and others with extensive knowledge of the information 
requirements of the Pell Grant and Caropus-Based programs have the 
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following specific advantages over those with no financial aid 
experience: 

• The ability to effectively interview financial aid 
officers, including being able to probe for meaningful 
responses » to recognize significant answers, and to 
know %ftien impressive sounding jargon is actually vague 
or meaningless. 

• The ability to collect data on individual students 
quickly and efficiently, given their familiarity with 
the record-keeping practices of institutions of higher 
education* 

• The ability to, collect reliable and complete data on 
individual students, given their knowledge of Federal 
financial aid regulations and the various financial aid 
forms and reports. 

• The ability, in debriefing sessions, formal reports, or 
the margins of questionnaires, to make knowledgeable 
observations about the workings' of financial aid pro- 
grams . 

Individuals with hands-on experience conducting Pell Grant vali- 
dation make particularly attractive candidates because much of 
the institutional site visit work involves abstracting data from 
the tax returns, bank statements, ED validation forms, and other 
verifying documentation collected by financial aid officers to 
comply with ED's validation regulations. 

In addition to experience in student financial aid, all 
applicants for the field representative positions should be 
judged on the following criteria; 

• Edu cation— a minimum of bachelor's degree required with 
ah advanced degree in an appropriate field desired 

• Intervie wing experience — prior experience on similar 
Studies with knowledge of research and survey methods 

« oral Communication Skills — speaks clearly, exhibits 
maturity, listens well 
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• Respon8ibilit.y — has held a position of responsibility, 
and is capable of working with little supervision 

• Endurance — in good physical condition, capable of 
%rarking at a fast steady pace, has travel experience 

Procedures 

1. Advertise for Financial Aid Professionals 

Advertise field representative openings in the Chronicle of 
Higher Education , the MASFAA Newsletter , and the Sunday edition 
of key large-city newspapers. For the Stage One study. Advanced 
Technology was able to attract nearly 100 resumes by using these 
advertising sources. Nearly all the applicants were financial 
aid professionals either between jobs or able to arrange a tem- 
porary leave from their current positions . A summary of the 
Stage One advertising/hiring results is shown in Figure 4-2. 

2. Interview by Phone 

Screen resumes and interview the most qualified applicants 
by phone. For a nationwide search, phone interviewing is effi- 
cient ana permits the employer to talk with a large number of 
candidates in a short period of time- 

3. Interview in Person 

Interview finalists in person and hire the most qualified. 
Task 4. 4 I Train Field Representatives 

The caliber of a training program is directly reflected in 
the quality of data obtained. The objective of training is to 
provide the institutional field staff with the exact knowledge 
required to collect precise data for the study. Through care- 
fully developed training, using methods that emphasize the 
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FIGURE 4-2 



RESULTS OP IMTERVIEWER ADVERTISING/HIRf NG 



Sources 

NASFAA Newsletter 



Resumes 
Received 

and 
Reviewed 



18 



FOR STAGE OWE QC STUDY 



Interviewed 
by 

Phone 



Interviewed 
in 
Person 



Offere<* 
Interviewing 
Position 



10 



Accepted 
Position 



Chronicle of H igher 
Education 35 12 



OSFA Bulletin 


0 


0 


0 


0 


0 


Boston Globe 


9 


1 


0 


0 


0 


Chicago Tribune 


3 


0 


0 


0 


0 


Houston Chronicle 


0 


0 


0 


0 


0 


Los Angeles Times 


0 


0 


0 


0 


0 


New York Times 


8 


3 


1 


1 


1 


Washington Post 


6 


3 


1 


1 


1 


Other 


7 


6 


5 


4 


4 


Cannot Determine 


9 


1 


0 


0 


0 


Totals 


95 


36 


19 


15* 


13 



NOTE: The following local organizations were also contacted to identify potential candidates in this area: 



1) NASFAA 6) George Washington University 

2) Prince Georges Cownunity College 6) NOVA - Alexandria 

3) University of Maryland 7) Joel Packer - U.S. Student Association 

4) Awjerican University 8) Applied Manageiient Sciences 
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importance of conBiatcncy, accuracy, and efficiency, thi« objec- 
tive can be accoropliehed. 

Due to both the complexity of the data collection and the 
importance of a well-trained field representative to the overall 
study, a minimum of five daya i« required to complete an effec- 
tive training program. If financial aid professionals or experi- 
enced DCPR program revie%irers are recruited, little time is needed 
to cover Federal student aid regulations and procedures. 
Instead, the training should focus on the purpose of the study, 
the purpose of survey research, interviewing techniques, and the 
precise procedures to be followed When abstracting data from 
student financial aid files* 
Procedures 

1. Develop Training Manuals 

Develop two manuals for the training program and for use by 
the xield representatives as a reference during the field work. 
One, the Training Manual , should describe the study and give pre- 
cise step-by-step procedures to be followed at each site, includ- 
ing precise information on %Aiere to locate certain student data 
on the campus^ Tk>w to record student file data, editing require- 
ments, travel logistics, and administrative details. With cer- 
tain modifications the manual used by Advanced Technology during 
Stage One could be revised for future Pell Grant error assess- 
ments . 

The second manual to be developed prior to the training pro- 
gram, the Question-by-Question Specifications , should include a 
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rspXlcation of Aach data coLldction instrument vlth specific 
information next to each item or question. This information 
should include but not be limited to U^e following: 

• Explanation of the purpose and intent of each item 

• Instruction for coding dollar and other values 

• Explanation for completing items and sections that may 
be confusing « complex, or potentially error prone 

• Warning about sXip patterns 

• Explanation for When to probe and ask for additional 
information driving an interview 

The Quest ion^by-Queation Specifications , in addition to being a 

training tool, is meant to be used as a guide when interviewing 

or collecting data from student files. The Question^by-Question 

Specifications manual used by Advanced Technology during Stage 

One could be revised and reused in a future error study. 

2, Contact Institutions to Arrange Field Practice 

Telephone institutions local to the training site and seek 

permission to conduct field practice on the next to last day of 

the training session (see step 4). Explain the purpose of the 

exercise to the financial aid administrator, the amount of time 

required for a practice interview (about one hour), and the 

amount of time to be spent collecting data from files (one to two 

hours is sufficient). Also, request photocopies of three or four 

financial aid files from one of the institutions. Ask that the 

institution delate all personal identifiers from the files. 

Those files are used during the training sessions for practice 

exercises in reviewing and abstracting data (see step 4). 
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minim pftoouK 

BE06 QUAim CONTROt STUDY 
AfiQBA 



Coffee and Danish 

emeral UalcM, Introtfiietlotts, and Ovarvltw of 
Training Wack - Hr. Joe Gartig and Ms. MeUnle 
Allnan, Advancad Tactmology 

Introduction^ Advanced Tedmology - Hr. Robert 
Darty, Advanced Tedmology 

Introduction md Background of BEOS Quality 
Control Study - Nr. Ernst Becker, Departnant 
of Education 

Quick Update on BEOS and Ca«pus-Based ProgrMs; 
Evolution of the Quality Control Data 
Collection Ittstrunents * Or. Alex Ratnofsky» 
Advanced Technology 

Explanation of Data Collectors' Tasks and 
Responsibilities - Nr. Joe Gertig, Advanced 
TechnologQr 

Break 

Fllw: Introduction to Interviewing ; 
Conducts the Institutional Interview With the 
FAO --Dr. Jill Bernstein, Advanced Technology 

lunch 

Review of Training Manuals - Melanle Alliwn 

Detailed Itas-by-ItOM Exaalnatlon of Institutional 
Interview Fom - Melanle Allian 

Deaonstratlon of Institutional Intarvelw - Jill 
Bernstein and Melanle Allnan 

Break 

Review Answers froe Deeonstratlon Interview; 
Group Practice: Beginning the Interview, 
Answering Respondent Questions, and Asking the 
Interview Questions - Jill Bernstein end Melanle 
Allnan 
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Tuesday. Marcti 24 



Coffee and Danlsli 

Role Piling of imitutlonal Interview with 
Trainees in Pairs 

Acting Out of Institutional Intertlew: Trainers 
Pi^ng Difficult Respondents with Trainees 
as Interrf ewers to Practice Probing and Handling 
Difficult Situations - ftelanle Allean, Joe 
Gertlg. an<! 'nr. Bill Ade 



Detailed Itea-by-Itee Exaart nation of Student Record 
Abstract and S|)ec1f1cs of Completlfig Abstracts - 
Nelanle Allnan 

Cowpletlon of Business Forms and Associated Paper- 
work 



Distribution of Hypothetical Student Financial Aid 
Record File; Perusal of Sample Forms Likely to be 
Found In Student Aid Files; and Review of Federal 
Tax Foms - Melanle Allean 

Practice Cosipleting a Student Record Abstract Using 
Data From Hypothetical Student Files 

Explanation of BEOG Alternate Disbursement System - 
Mr. Roy Watson, Advanced Technology 



Practice Co«plet1ng a Second Student Record 
Abstract a 

Recap of First Two Days of Training 



Break 



Lunch 



Break 




Student Record Abstract 
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Wg^ngfday. Harch 25 



8:15 - 8:30 CoffM and Danish 

8:30 - 9:15 Ravlcw of AntMers for Tuasday Evanlng Asslgnnent 

- Nalanle Allmn 

9:15-10:15 Role Flaying of Second Institutional Interview in 

Pairs 

10:15 - 10:30 Discussion and Practice with Corrections Control 

Group Foms * Melanle Allain 

10:3e - 10:45 Break 

10:45 - 11:30 Role Playing of Resolution/Exit Interview with FAD 

- Melanle Allmn and Bill Ade 

11:30 - 12:15 Issuing of Cash Advances; Trip to Bank - Joe Gertig 

12:15 - 1:00 Lunch 

1:00 - 2:00 Explanation of Shipping and Receiving of Data Col- 

lection Materials, Editing, and Post- Interview 
Proce<^res - Bill Ade 

2:00 - 2:30 Presentation by Data Analysts Explaining What Hap- 

pens to Completed Data Collection Forms Upon 
Return to Project Office: Log-In of Forms, 
Interview Verification, Coding, Keypunching, and 
Data Processing - Bill Ade, Jill Bernstein, and 
Jennifer Zlii nwrwa n 



2:30 - 3:30 Explanation of Travel Arangeisents— Cash Advances, 

Hotel Reservations, Airline Tickets, Car Rental 
Rental Proceifcires and Expense Reports; Discus- 
sion of Anticipated Problems - Joe Gertig and 
Bin Ade 

3:30 - 3:45 Break 

3:45 - 4:00 Explanation of Thursday, March 26, Field Practice 

- Melanle Allnan 

4:00 - 5:00 Practice Completing a Fourth Student Record 

Abstract 
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TlHirsday. Warch 26 



Field Practice at the Folloiiinj Area Institutions 



THnl^ Collefe 
Washington, O.C* 



Goiicher College 
Towsont M) 



St. John's College 
Annapolis, MD 



Oundalk .OMUunlty College 
Dundalk, M) 



'^rede*-*''*' Zcmunity College 
rrederick, «) 



Shenandoah College and Conservatory 

of Husic 
Winchester, YA 



Capitol Institute of Technology 
Kensington, MD 
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Friday, March 27* . 



8:15 - 8:30 CoffM and Danlsii 

8:30 - 9:00 Photo UD.t Taken 

9:00 - 10:00 Detallfid Rerfew of Previous Hiy Field Practice-- 

Oisaissions of Experiences, Answers to Trainee 
QuestloiiSy Advice on How to Handle Probleai 
Situations 

10:00 - 10:15 Smmry - Dr. Ted Bartell 

10:15 - 11:00 Tour of SSOC; Confimation Calls Made to First 

Week Institutions 

11:00 - 1:00 Lundi 

Hotel Checkout 

1:00 - 2:00 Recap of Training Week; Completion of Paperwork; 

Answering of Questions 



* Friday training will be held at the Westpark Hotel 
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Practice sit* visits can toe arranged at the same time institu- 
tions in the regular sample are being contacted (see Task 4.2). 

3. Mail Training Manual to Field Representatives 

Once hired, field representatives should be notified as to 
the time and location of the training session. Also, mail the 
Training Manual to each representative with instructions to 
review it before the first day of training. Those vftio study the 
manual will begin the formal trainizig session with a basic under- 
standing of the goals of the study of interviewing techniques, 
and of the use of the data collection instruments* 

4. Conduct Training Session 

The agenda for the five day training session conducted by 
Advanced Technology during Stage One is shovm in Figure 4-3. 
This agenda proved to be effective, and it is recommended that it 
be uscKi as a guideline for planning future sessions . There are 
five basic training techniques that should be used during the 
training session: interactive lecture, film presentation, role 
playing, exercises, and field practice. The following describes 
each of these techniques amd gives axeunples of the material that 
can be presented with each technique. 

Interactive Lecture . This technique is typically used for 
explaining the purpose of the study, reviewing pertinent student 
aid regulations and procedures, explaining of basic interviewing 
techniques, describing the field procedures and other administra- 
tive details, and presenting the basic concepts of the data col- 
lection instruments. Also, the lecturer should lead the trainees 
through the questionnaire by calling on trainees to act the role 
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of interviawer \^ile the lecturer plays the financial aid officer 
or some other institutional administrator. Each trainee records 
the responses in his or heij blank copy of the questionnaire vihLle 
a member of the training staff records responses on a transpar- 
ency projected on a screen at the front of the training room* 
Trainees are reminded after each question to check their 
recording against the screen. 

Film Presentation * For the Stage One training session « 
Advanced Technology showed a film on general interviewing tech- 
niques Which covered such topics as gaining cooperation, main- 
taining rapport, asking questions, and probing responses. Film 
presentations are particularly helpful since they capture and 
hold the attention of the trainee better than a lecture does . 

Role Playing . Once the lecturer has explained each item on 
the questionnaire, it is important that each trainee gain prac- 
tice conducting mock interviews. The trainees break up into 
pairs. Within each pair, one trainee takes the role of inter- 
viewer While the other plays a financial aid officer who responds 
to the questions using a prepared script. Trainees playing the 
respondent role ar«^ cautioned not to coach the ones playing 
interviewer, but, as much as possible, to create a true inter- 
viewing situation. Eadh pair should conduct at least three mock 
interviews with the script for the initial interview being rela- 
tively straightforward and the scripts foj^ the last two con- 
taining difficult responses. 

Exercises . The trainees should gain substantial experience 
in reviewing hypothetical student aid files and recording data in 
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the Student Record Abe tract during the training sees ion. Sample 
financial aid files are compiled prior to the first day of the 
training session (step 2)^ During the actual sessions or as an 
overnight assignment/ the trainees complete the sample record 
abstracts. The completed abstracts are reviewed individually by 
the trainees, and any problems or major errors are discussed with 
the groups 

Field Practice . One day of the training session should be 
allocated to field practice at a local institution. The trainees 
should be divided into groups with a maximum of three to a group. 
Each group should be accompanied by a training supervisor. The 
practice site visit is basically an enactment of a typical full 
day in the field. It includes an introduction to the study, an 
interview of the financial aid officer, a review of student 
files, a review with the financial aid officer of apparent errors 
found in the files, and a field edit of the data collection 
instruments. The following day, the training group discusses 
their experiences, and the trainees answer any questions raised. 
Based on the field practice, the supervisors evaluate each 
trainee *s preparedness for entering the field and hold special 
conferences for any in whom knowledge gaps or special problems 
are detected. 

Task 4.5; Conduct Site Visits 

Fourteen separate activities must be completed for each site 
visits as shown in Figure 4-4. Resolution of discrepancies, a 
fifteenth activity, will be required at many institutions. 
Depending on the sampling procedure, used, the number of records 
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FIGURE 4-4 
INSTITUTIONAL DATA COLLECTION CYCLE 

BEST copy AVAILABLE 



to be reviewed per institution may vary considerably, conse* 
quently, the scale of the visit may vary from less than a day 
to seven days at any one institution « Recordkeeping systems at 
institutions also vary, and these, too, will affect the length 
and complexity of the visit. Nevertheless, each of the steps in 
Figure 4-4 must be carried out to some extent at every institu- 
tion. 

Procedures 

1. Prepare for the Site Visit 

The site visitor should call the financial aid office at 
least two days (but no more than a week) in advance to confirm 
the appointment, travel directions, and parking and chec':-in 
procedures. On occasion^ site visitors may fall behind schedule 
and have to reschedule a visit. Kescheduling should only be done 
by field personnel if it will not affect other institutions later 
in their schedules. Otherwise, rescheduling should be done by 
central project staff- 

Before arriving at the site, the data collector must check 
to be sure to have all the necessary forms and materials for the 
visit. 

2 . Meet with Financial Aid Director 

The meeting with the financial aid director includes three 
activities: introducing the study, conducting the interview, and 
scheduling the exit interview* 

2a* Introduce the Study. The data collector should present 
his credentials and describe the nature and purpose of the study, 
particularly the site visit component. Althouih the visit will 



have been preceded t»y letters and telephone calle# many financial 
aid directors will have only a vague idea of the study prior to 
the visit. The data collector should also describe for the 
financial aid director the steps he will follow after the inter- 
view and the records he will need to review. 

2b. Conduct the Interview. The interview should be con- 
ducted by following the formal questionnaire as closely as 
possible- During the training week, the field representatives 
are given extensive instruction in conducting the interview. 
They are told to observe the following basic guidelines when 
asking questions and recording responses: 

• Remain neutral 

• Ask all questions exactly as worded 

• Discourage unrelated conversation 

• Ask respondents to enlarge or clarify answers when 
necessary 

• Record verbatim the resporident's questions 

At some institutions ^ especially large ones, the financial 
aid director may prefer to have Pell Grant questions answered by 
a staff specialist in that program, either by having a joint 
interview or by referring the data collector to that person. In 
some cases, the director may not even be present, having left the 
entire visit in the hands of a subordinate because of scheduling 
problems. Someone other than the financial aid director can be 
interviewed so long as the substitute can give authoritative 
answers about institutional policy and practices. In previous 
field work, alternate interviewees have been scheduled for some 
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institutiona during th« telephoning required to draw up the 
master achedule* In some cases, no one on the financial aid 
staff may be able to answer some of the interview questions, and 
the bursar may havs to be questioned about fiscal operations and 
refund and repayment policy, and the registrar on procedures used 
to check enrollment status. 

2c. Schedule Exit Interview, ^pon completion of the formal 



q lestionnaire, the visitor should arrange to meet with the finan- 



cial aid director or someone on the staff to discuss any discrep- 
ancies found in the records of individual students. Also at this 
time, arrangements must be made for access to the student aid 
files. Especially at smaller institutions, the director may be 
uncomfortable with having an outsider reviewing files, and so may 
want to be present during the file review and pull the individual 
files personally. At larger schools, this function will usually 
be delegated to a subordinate professional or a secretary or 
clerk. 

3. Complete File Abstracts 

Assuming that the sample has already been drawn (in a pre- 
vious visit or from lists furnished by the school before the 
visit), the site visitors can present a list of the students 
sampled at the institution and ask to see their financial aid 
files and SARs . The type of files available will vary tremend- 
ously, small schools are more likely to have manual filing 
systems where all the records for each student are kept in a 
folder in alphabetical order. At such schools, the relevant 
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filea can simply b« pulled and reviewed? some inetitutione may 
even be willing just to show the visitor the file and let him 
pull the files himself. Many institutions have some or all of 
the relevant data in computer files. There, a staff member will 
have to show the site visitor how to access specific files and 
how to interpret the information which appears on a monitor; or, 
a staff member may have to requert a printout of the relevant 
data for the sample students and then explain the printout format 
to the visitor after it is delivered. 

Hard copies of SARs and required documentation should be 
available at all schools. Ho'ever, these and other records may 
be most readily available on microform. These, too, will nor- 
mally need some explanation from financial aid office personnel 
before they can be used by the site visitor. 

Many financial aid offices will have disbursement records 
and documentation of enrollment status and satisfactory academic 
progress, or copies of the required records, in their student aid 
files. In other cases, these records may be part of the elec- 
tronic file accessible on the terminals in the financial aid 
office. At some schools, however, disbursen»nt records will be 
available only at the bursar's or treasurer's office or office of 
student accounts, and enrollment and academic progress records 
will be available only at the registrar's office. Although the 
telephone protocol used to schedule the visits should have 
included questions designed to determine whether all the record* 
are in one place, the answers may be arabiguous or wrong. As soon 
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as tha site visitor dstcmiines that soms records are not in the 
financial aid office, he should ask for arrangements to be made 

t 

to obtain access to ths other offices, so they will have at least 
a couple hours* notice of his visit* Especially cooperative 
financial aid directors send memos to the bursar and registrar 
infonuing them of the date and legitimacy of the visit and of the 
possibility that th*^«ir records oay have to be consulted* At 
small institutions, the "director of financial aid" may be a 
vice-president with authority for student accounts and academic 
records as well as financial aid; once his cooperation with the 
study has been secured, the other offices pose no problem. 

The field representative must place a notice in each 
reviewed file stating the purpose and the date of the data col- 
lection. This notice satisfies the conditions of the privacy 
regulations . 

4. Complete Control Group Forms 

While reviewing the student files to complete the SRA«, the 
field representative should record information from the SAR onto 
the Corrections Control Group (CCG) form. 

5. Conduct Exit Interview 

Before leaving a site, the field representative must conduct 
a brief exit interview with the financial aid officer, the pur- 
pose of which is to thank the aid officer for his or her cooper- 
ation, and to discuss discrepancies found in the student files to 
learn whether the aid officer can offer a logical explanation for 
what on the surface appears to be an error or violation of Pell 
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Grant regulation*. Tha fiald rapr«aantative muat usa tact in 
seeking this explanation, in order to reassure the financial aid 
officer, if necessary, that his or her response will be kept in 
confidence and used for national estimates only. 

6. Edit Completed Instruments 

At the end of each work day, the field representative must 
edit all cranpleted ir»truraent8 for possible omissions, inconsis- 
tencies, illegible handwriting^ or misplaced codes, if the field 
representative is scheduled to ret^im to the institution the fol- 
lowing day, he or she can take advantage of .this opportunity to 
clarify or retrieve any missing information. Otlverwise, the 
field representatives do not recontact the institution to 
retrieve data. Instead, they vrrite notes next to the items in 

o 

question and, once the instrument has been received, senior staff 
roerabers at the project office attempt to retrieve the incorrect 
or missing data by telephone (see Task 4.7s "Follow up by 
Telephone" ) . 

7. Complete Transmittal Form 

The transmittal form lists all the student record abstracts, 
interview forms, and control group fomm being sent to the pro- 
ject office. It is important to list everything being sent so 
that the receiving clerk can be sure that nothing was lost in the 
mail. It also gives the data collector ar opportunity to make 
sure he is sending in all the required forms. 
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8* " Bundle Data and Insert in Pre-Addressed Mailer 

Pr e*^addr eased r padded envelopes are provided to the data 
collectors to ensure that data are correctly addressed and pro- 
tected during shipment* 
9* Seal^ Tape^ and^Mall Data Package 

First class mail is used to transmit data because of the 
widespread availability of post office and drop boxes* Special 
services (registered mail^ express mail^ special delivery) do not 
add enough security or speed to the delivery to be worth the 
cost. In previous data collections, no data have been lost or 
inordinately delayed through the use of ordinary first class 
mail- 

The field representative should mail completed instruments 
to the project office every two or three days. Waiting longer 
than three days increases the risk that instruments will be lost 
in the field. Instruments should be sent in heavy-duty envelopes 
accompanied by a transmittal form detailing the contents of the 
mailing . 

10 • Record MailingL Date and Location in Notebook 

Recording the date and location of mailing provides a useful 

record in case of any disputes with either the Postal Service or 

the data collector about delay or loss of forms. 

1 1 ♦ Cal 1 Supervisors 

Data collectors should call the field supervisor once a week 

to discuss the progress of the data collection and any problems 

or questions encountered in the field* This call will also ena^ 

ble the supervisor to regularly discuss any procedural or coding 




errors by the data collector, or any vague, undocumented, or 
unallowable field expenses* Critical problems should be resolved 
by an immediate phone call from either the supervisor or the data 
collector, and not left to the weekly call. 
Task 4.6: Implement Quality Control and Supervision Plan 

To ensure that the data collected for the error study are 
accurate, timely, and obtained at minimum cost, a supervision and 
quality control plan must be implemented at the start of the 
field period. Communication with the field representatives will 
take place through periodic memoranda and through scheduled week- 
ly telephone calls from the field staff to the project office. 
In addition to quality control measures such as the field editing 
(discussed in Task 4.5) and the manual and machine editing con- 
ducted in the project office (discussed in Chapter 5), a sample 
of completed instruments must be randomly selected and validated 
by the project office staff to ensure the reliability of the 
collected data. In addition, field representatives should be 
monitored in person to ensure that they are following all pro- 
cedures properly and recording the correct data. 
Procedures 

1* Send Periodic Memoranda 

Send memoranda to the field representatives as often as 
nepessary throughout the field period to communicate updates to 
existing procedures or implementation of new procedures. 
2 . Establish Telephone Schedules 

A separate telephone line with an 800 number and a. recording 
device should be installed in the project office. At least once 



a week on a scheduled day and time the field representatives 



should be required to call the project office. These weekly 
calls can serve three purposes: 

• Monitoring of Data Collection . During these calls a 
project staff member can answer questions on interview- 
ing technique / instrument, administrative, coding con- 
vention, and so on. 

• Changes in Interviev^ Travel/ and Accowroodation 
Arrangements # There may be unforeseen changes to the 
field representative's itinerary initiated by financial 
aid officers, hotels, or airlines. In all cases, the 
field representative should be rcK}uired to report 
schedule changes to the project office. If the pro- 
gress of a particular field representative lags due to 
unforeseen data collection or travel problems (e.g., 
sickness, bad weather)/ a project staff member can use 
the weekly telephone call to discuss ways of resolving 
the schedule problem with the field representative. 
Sites of field representatives whose progress is lag-* 
ging can often be reassigned to others who are ah'^ad of 
schedule * 

m Clarification of Routine Business Matters . Issues 

related to expense reports, travel advances, paychecks, 
and so on can also be discussed during the weekly 
calls . 

The field representatives should be encouraged to call the 
project office nrare frequently than the required weekly call. A 
recording device should be installed to take messages after 
business hours. 

3 . Validate the Field Representative's Work 

During the field period, verify that the institutional site 
visits are being conducted according to correct procedures. Sel- 
ect one recipient at random from each institution in the sample. 
Telephone the financial aid administrator to confirm that the 
student records were, in fact, inspected, and that the conduct 

/ 

( 

r 
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of the field representative was appropriate* In addition, the 
financial aid officer should be asked to verify two questions 
from the Institutional Questionnaire (XQ) and one from the record 
abstract of the selected recipient. The items selected for vali- 
dation should be those unlikely to have changed in fact or in the 
financial aid officer's perception between the time of the site 
visit and the validation call* For the Stage One study^ Advanced 
Technology asked "What Types of Documentation of Income Are Rou^ 
tinely Collected by Your Financial Aid Office?** and "How Fre- 
quently Are Payments Disbursed to Eligible Recipients?" from the 
interview, and **What Is the Recipient's SAI?** from the record 
abstract • If a financial aid officer makes a negative' evaluation 
or a discrepancy is found between the field representative's 
findings and the validator's findings, the field representative 
should be contacted immediately for an explanation* 
4. Observe Field Representatives On-Site 

Monitor each field representative on-site at least once 
during the field period • The monitoring visits should occur 
during the first two weeks so that any problems in field proced- 
ures can be found and corrected early. During the site visit, 
observe the field representative's interviewing technique, pro- 
fessional manner, thoroughness, and accuracy. Review several 
completed file reviews thoroughly. Discuss any problems found • 
Task 4.7; Follow up by Telephone 

Field representatives may send instruments to the project 
office with critical items left blank or with data that are 
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inconsistent or illogical. Missing and inadequate data can be 
collected efficiently at low cost by telephoning the field rep- 
resentative or the institution* Experience suggests that, 
although extremely important, this is not a time-consuming task. 
In Stage One, only 5 institutions out of a total sample of 305 
were contacted to retrieve data. 
Procedures 

1 . Telephone Field Representative or Institution 

Once the missing or inadequate data are found (usually 
during the manual or machine editing stages), call the field 
representative or institution. 

2. Code Instrument 

Code the instrument based on the new data collected in the 
telephone conversation. ^ 
Tasic 4.8; Debrief Field Representatives 

Data collection for the institutional compo .nt of the error 
study should not end with the last site visit. The field repre- 
sentatives, particularly if they are financial axd professionals, 
will have many observations and recommendations regarding error 
in the Pell Grant program. 
Procedures 

1. Plan One-Day Debriefing Session 

Develop an agenda for a one-day panel discussion among field 
representatives and project analysts* The debriefing agenda used 
by Advanced Technology during Stage One is shown in Figure 4-5. 
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2. Conduct Debriefing Sgssion 

During the day-long session, discuss significant problems 
with the Pell Grant delivery system and quality control proce- 
dures being used et institutions. Also, discuss ways in which 
the institutional data collection could be improved for the next 
study. 
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CHAPTER 5 
DATA PREPARATION SPECIFICATIONS 



OVERVIEW 

The purpose of the data preparation system is to compile. 
Verify / convert, and organize raw survey data onto a computerized 
data base for analysis* The major objectives of the system are 
to: 

Ensure the timely and efficient processing of the 
incoming raw data 

Ensure the complet3ness and accuracy ox the survey data 

Maintain control of the status of all data collection 
instruments to prevent them from being lost or mis-- 
placed 

Ensure the transcription accuracy of coders and data 
entry staff 

Maintain the confidentiality of the data 

Ensure that "cleaned" data files are successfully 
merged to form a single file of "best verified" data 

Technical specifications for nweting these objectives are set 
forth in this chapter task by task and step by step* As in Chap- 
ter 4, the descriptions of each task follow a set format: first/ 
a discussion of the purpose of the task and any important issues 
that need consideration and/ second/ a step-by-step list of pro- 
cedures* Figure 5*1 presents an overview of the tasks and pro- 
cedures described in this chapter. 

In general < the procedures set forth in this chapter i-pply 
regardless of the type of data collection instrument or hard copy 
form that will be processed. For example, 30 page interview 
questionnaires with many open-ended questions must be scanned/ 
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cod«d, adited and keyed, in the same basic way aa 2-page 1040 tax 
returns . 

Schedule Considerations 

Data preparation should begin once the data start arriving 
from the field rather than after a complete set of data has been 
collected, for two reasons. First, analysis of findings can 
begin and be completed much sooner than otherwise. Second, 
telephone follow-up of respondents or interviewers who have sent 
incomplete or inadequate data to the home office is mora success- 
ful the closer it is conducted to the time of the interview or 
the data collection. For example, if data preparation is done on 
a continuous and efficient basis, an important omission on a 
questionnaire sent by a field interviewer can be discovered 
within a week after the interview* When contacted,- that inter- 
viewer will likely remember the interview and will be able to 
provide the information over the telephone. If not, the inter- 
viewer might still be in the vicinity where the interview was 
conducted and will be able to recontact the respondent. 
Security and Confidentiality 

The organization responsible for conducting an annual 
assessment of Pell Grant error must be firmly committed to pro- 
tecting the confidentiality and privacy of individual data col- 
lected from students and parents. All data preparation activi- 
ties should be conducted in accordance with the spirit and letter 
of the Buckley Amendment and the Privacy Act of 1974. The 
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following steps should b« taken to safeguard the security and 
confidentiality of the datas 

• Receipt control^ coding, editing, and keying operations 
should all be conducted in lockabls rooms accessible 
only to project staff. Cleaning personnel should only 
be allo%#ed to enter the rooms When one of the project 
staff is also there. 

• Data collection instruments must be maintained in lock- 
able file cabinets in these rooms . 

• KL\ staff members Who handle individual respondent data 
should sign a confidentiality pledge. 

• All the necessary documentation of cc»nputer files 
should be kept in a file accessible only by authorized 
project staff analysts. 

• If any ccmiputer file needs to be processed' outsits the 
hcmie office, rva personal identification should appear 
on a respondent data record, instead a system of files 
should be formed with File 1 containing personal iden- 
tification (name, address, and telephone number) for 
each sampled recipient. Pile 2 containing the responses 
for eac^ participant, an4 File 3 containing information 
which links File 1 to Pile 2. In such a system there 
can be no link between respondent data and personal 
information without the presence of File 3. 

TASK 1; RECEIVE DATA 

Central to the data preparation effort, particularly v<hen a 
large number of data collection instruments is involved, is a 
well-defined system to be used by home office staff in the 
receipt, logging, and routing of all instruments. Procedures 
must be instituted to maintain control of the status of each 
instrument frcsn the time received at the home office to the time 
the data are entered onto a computer data base. 

It is highly recommended that the receipt system make use of 
a ccmiputer ized file of respondents, particularly %»ftien the sample 
is large and the data collection requires a telephone follow-up 
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of nonr«apondents . Th« raapondent fil«, created during th« 
sajuplii^g «tage, should include at a rainirauai the names « addresses, 
and Social Security numbers of sampled students and parents and 
the names and addresses of sau^led institutions. The file can be 
used to: 

• Generate a log for cataloging data instruments as they 
are returned from the field 

• Produce up-to^ate reports on the progress of the data 
collection and data preparation efforts 

• Identify nonrespondeats 
Procedures 

1. Record on Master Control Log 

As instruments arrive in the home office, a log-in cleric 
should record the date received and his or her initials on the 
master control log. This log, generated from the computerized 
respondent file, should include the name and address of each 
respondent in study identifier order- Space should be provided 
on the log for updated information on names and addresses. 
Figure 5-2 shows the log used by Westat during the Stage One 
study. The master control log's usefulness is demonstrated when 
someone wants to know a certain instrument's Whereabouts. To gat 
this information, simply look up that instrument's identifier in 
the master control log: absence of any entries indicates the 
instrument has not been received. 

2. Log onto Respondent Control File 

Once instruments have been recorded on the master control 
log, the clerk should enter the fact that the instrument was 
received and any address updates onto the control file. In the 
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stage One study, We^tat ueed an optical character recognition 
(OCR) Wand Reader which scanned each instrument and transferred 
information to the control file. A card update approach can be 
used, although it is nK3re time consuming and less efficient. 
3. Scan Edit 

After logging in, the clerk must review the instrument to 
determine if it is^ sufficiently complete to continue to be pro- 
cessed* The instrument should be considered incomplete if more 
than 10 percent of the questions or items are blank or if any 
critical piece of information was not provided. In the Stage One 
study critical items were those items needed to calculate total 
error, institutional error, total student error, and application 
item error. Incomplete instruments should be set aside for tele- 
phone -.ollow-up (see Chapter 4); complete instruments should con- 
tinue to be processed. 
4- Batch 

The clerk should then batch instruments into groups of 
manageable size (norroally 10-50 instruments per batch) and attach 
a transmittal sheet identifying the contents of the batch. The 
batch should be given an identifying number. Bat. ing facili- 
tates the handling of a large number of instruments and minimizes 
the risk of losing and misplacing instrioments . 
5. Record on Master Control log and Batch Control Log 

Ortce a batch has been formed/ the clerk should record its 
I.D. for each of its instruments* on the master control log and 
then make an initial entry in a batch control log, A batch 
control log, an example of which is shown in Figure 5-3, should 
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ba used to trace the path of each batch of instruments through 
all processing steps « A batch's line entry should be initialed 
when the batch is formed and then updated (initials/ dates , 
remarks) as the batch enters or exits a processing step. 

Using the master d^ontrol log and the batch control log, the 
exact status and locati<i>n of an individual instrument can be 
found as follOMS: 

• Locate the instrument's I*D. number in the master con- 
trol log. Find the number of the batch containing the 
instruments 

• Locate the batch number in the batch control log* 
Determine the physical location of the batch* 

TASK 2; EDIT AND CODE 

Once the data collection instruments have been received and 
logged in, they must be thoroughly reviewed for completeness and 
accuracy and their responses must be translated into numerical 
codes* In reviewing and coding the instruments / the coding staff 
can make decisions that have a substantial impact on the analyt- 
ical findings* Therefore, it is very important that; 

• A detailed, easy-to-use, coding and editing specifica- 
tions manual is developed to guide coders/ to act as a 
reference for programmers and amalysts, and to provide 
documentation of the data files - 

• Coders are thoroughly trained 

The coding and editing specifications msuiual must contain the 
following ; 

• Codes for all possible responses, including codes for 
missing data such as "response not ascertained,'* "inap- 
plicable" (legitimately skipped), and ''don't know" 

• General editing instructions including the logical 
range of values precoded by the interviewers or 
respondent 
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• Clear delineation of skip patterns 

• Variable names for all items to be coded/edited 

The text of the manual should be Xept on a vrc>rd processing sys- 
tem so that new codes can be quickly added. Figure 5-4, a 
selected page from the student interview manual used by Westat 
during Stage One, shows an exemplary format for a coding and 
editing specifications manual. Westat 's and Advanced Technol- 
ogy's Stage One raamuals could easily be modified for an ongoing 
annual assessment of Pell Grant error once a decision is made on 
the data that will be collected. 

The coding staff must be thoroughly trained in the use of 
the manual, the general data preparation procedures, and the 
regulations and procedures of the Pell Grant progrzun. One day is 
normally a sufficient amount of time for training. The training 
session should include both an itera-by-item explanation of 
coding/editing specifications with attention paid to potentially 
error-prone areas and the coding of several practice cases con- 
taining specially selected response patterns. These practice 
cases should be carefully reviewed by a senior staff member. 
Coders with unacceptable practice performance should be given 
additional training. 
Procedures 
1 • Code 

Assign coders work by coding batch, and require them to com- 
plete the coding of one batch before beginning work on another. 
When a coder receives a batch, the batch's assignment must be 
noted in the batch control log. 
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Due to questions in the data collection instruments Which 
are open-ended in format/ coders will often not be able to code 
responses with any of the predetermined list of codes in the 
editing and coding specif ications manual- In such cases^ the 
coder must record the response and forward it to a senior staff 
member designated as the coding supervisor who, if necessary, 
constructs a new code* Coding decisions must remain consistent • 
Therefore, the coding supervisor should serve as the focal point 
of all decisions • It is important that all decisions be recorded 
on a log, an example of which is showi in Figure 5-5, A decision 
log can aid the coding supervisor in making consistent decisions 
throughout the coding process. It also can provide information 
to revise the data collection instruments and the coding and 
editing specifications manuail for the next data collection 
effort* At least once a day, the coding staff should be informed 
of all coding decisions; coders should then be responsible for 
updating their manuals. 
2. Edit 

All instruments during coding must be reviewed for complete- 
ness and accuracy. Coders should check skip patterns to see that 
they are followed correctly, check responses for legibility and 
relevance, and check the consistency and logic of all data* 
Although coders scan all items on the instruments , certain ques- 
tions that are found to be error-prone should be given particular 
attention. When coders find erroneous skips, illegible answers, 
or illogical responses, they should document the problem and 
refer the case to the coding supervisor. 
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Often missing, incorrect, or illegible data can be recon- 
structed on the basis of responses to associated questions or 
notes by the respondent or interviewer in the margins of the 
instrument form. If data which are critical for measuring error 
cannot be reconstructed, the instrument should be set aside for 
telephone follow-up (see Chapter 4). 
3 . Conduct Quality Control Check 

High perfoxrroance standards must be maintaincKi for all coders 
through sampled inspection of each coder's work. At the begin- 
ning of the process, the coding supervisor should inspect all 
ins rumen ts in each coder's completed batch. Once a coder 
attains an error level below .1 percent (for example, one error 
every 10 100-item instruments), verification can be cut back and 
performed on no more than 10 percent of the coder's subsequent 
work- Verification sheets showing number and types of errors 
should be maintained by the coding supervisor and copies distri- 
buted to the coders- Whenever a coder's error rate exceeds the 
.1 percent standard, or a pattern of errors appears, verification 
of that coder's work should be increased, and feedback or 
retraining on specific procedures given. 

TASK KEY ENTER 

Westat's and Advanced Technology's experience in entering 
large volumes of data has proven batch entry to be more economi- 
cal, efficient, and effective than on-line entry- Data laay be 
entered from key to card, key to disk, key to disk to tape, or 
key to tape- The data processing environment of the organization 
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which conducts th« annual arror aasessmant (ED or contractor) 
will dictate which data antry approach is most advantageous . 

The data antry staff must be provided with keying instruc- 
tions which include the record (card) layouts for each data col- 
lection instrument, the data field locations, the allowable con- 
tent for each data field (alphabetic, numeric, zero fill, and 
skips), and valid field ranges. 

The data entry machinery roust allow for simple editing of 
data items to ensure keypunching accuracy. These edits should 
perform alphabetic and numeric field checks and field range 
checks. The key entry system should also have the capability of 
generating reports for the data entry manager and detailing pro- 
duction and error rates accumulated by each key entry operator. 
This report will allow the manager to screen for systematic 
errors and to control tightly the quality of all data keyed into 
the system. 
Procedures 

1, Key Enter 

Keying should be performed in batches, transferring data 
directly from instruments to cards, disk, or tape. Once a keying 
operator receives a batch, the assignment should be recorded in 
the batch control log. Keypunch errors caught by the key entry 
edits must be corrected immediately by the keypunchers . 

2. Key Verify 

Each batch of keyed data must be 100 percent key verified 
against the associated instruments by an operator other than the 
one who originally keyed in the data. The data entry manager 
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should maintain a 'record of errors for each keying operator. If 
an operator's error rate exceeds an unacceptable level/ the 
manager, should give feedback or retraining. 

TASK 4; MACHINE EDIT AND UPDATE 

When the data are returned from keypunching they must be 
processed through editing software to verify their correctness 
and. ccsnpleteness . The editing system performs range checks on 
individual items ar^ cross checks on related items to ensure that 
all fields contain data within valid ranges and consistent adross 
data fields • 

All editing software should be developed and in place by the 
" time the first data are returned fran keypunching • The program- 
mers designing the software must have some input into the forms 
design stage of the task so that the best possible record layouts 
can be designed* The programmers should also be present when 
coding convencions are established in order to guarantee that the 
coding' methods chosen do not prevent efficient editing.- After 
coding conventions are established^ all the software can be 
designed and tested ihe major portion of the editing should be 
done by a COBOL program designed to edit 80 byte records 
according to specifications provided to it through a group of 
parameter cards* These parameter cards must be written for each 
type of data to be edited, e.g., parent, student or institution 
data, and must specify range checks and cross-field checks to be 
performed on data items. 



Editing software must be changed aach time a new code is 
established by the coding supervisor. It should be the responsi- 
bility of the coding supervisor to keep the prograunraers informed 
of all code changes as soon as they are established. 

Data that include multiple records per case require addi- 
tional editing to verify that all data are present in complete 
sets. A short EASYTRIEVE set check prograuu will accomplish this. 
This program can be written at any point after the forms design 
stage . 

Procedures 
■ - — 

Note that all editing software described here has been 
developed by Advanced Technology. With certain modifications and 
enhancements, the COBOL and EASYTRIEVE existing programs could be 
reused for subsequent error studies. 
1. Run Edit Program on Keyed Data 

The data should be forwarded in batches from the key entry 
site with documentation identifying the type of data and the num- 
ber of cases or records contained in each batch. The data should 
be used as they are batched unless batches contain a small number 
of records (e.g., less than 200) in which case several batches of 
the same type data will be combined and run together. 

Input data into an IBM sort progreim and sort first by I.D. 
and second by card number. This is the order in which the tape 
will be processed throughout the editing cycle and it is also the 
order of the final tape. The output to the sort will be a sorted 
tape . 
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Input the sorted tape into the appropriate COBOL editor 
program^ The invalid data will be written to disk and to the 
printer* The valid data will be written to tape* The printed 
output will identify each error by field name and print the 
entire data record • Control totals will also be butput and mu3t 
be checked to ensure that all data records have been processed 
and are accounted for on either the valid or invalid data files* 
2. Resolve Edit Failures 

Pass the printed output to the coders to be reviewed and 
corrected* Coders must locate the original instrument for each 
record in error and make corrections on the printed output • If 
the error cannot be corrected by reviewing the instrument/ the 
form must be forwarded for telephone follow-up • If the correct 
value for the data item in error cannot be obtained, the coders 
should designate it as missing data. When all errors for one 
batch are resolved the printed output must be returned so that 
the file can be updated. 

3- Update File 

Review the corrected output to ensure that all errors have 
been resolved and will not be rejected again by the COBOL editor* 
Corrections must then be made to the error records on the disk 
file* This can be done on the CRT by data entry staff. 

4- Repeat Edit Process Until File Is "Clean*' 

Input the corrected disk file into the COBOL editor* If any 
errors are found in this run, the correction process must be 
repeated until there is no invalid data found by the COBOL 
editor . 
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When the data frcxa the batch are error free they will be 
located on several different tapes. These tapes must be merged 
together. Since all tapes are sorted in the same order* they can 

merged by using an IBM merge. The output of this merge will 
be one tape with all the clean data from one batch. All 
intermediate tapes and disks should be purged at this point. 
5. R\m Edit Program to Ensure Cong>letenesa of Data 

Input all "clean" data tapes that contain multiple records 
per case into the EASYTRIEVE set check program. This program 
will output a printed list of all data records that are part of 
inccroplete sets of data. These errors must be resolved in the 
same way as tHe errors frcxa the COBOL editor. To correct set 
check errors, an EASYTRIEVE program must be written and executed - 
The output will be a tape containing the final cleaned and com- 
plete data for the batch. This tape should be run through the 
set check program again to verify its correctness. 

Store the final tape from each batch until it is needed for 
analysis or until all batches of data for one type of data are 
clean . 

TASK 5: REFORMAT FILES FOR ANALYSIS PACKAGE 

The particular analysis package to be used in the study will 
determine what type of data file is necessary- Past experience 
suggests that the Statistical Analysis System (SAS) is the most 
efficient and flexible software program for this type of study. 
SAS, for example, is considerably more flexible than other pro- 
grams for merging records which contain varying types and amounts 
cl information into a single data base. 
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The data at this point are stored on IBM operating system 
(OS) files. For use in SAS it should be reformatted into a SAS 
file. Although the data can be read directly from the OS files, 
reformatting the files allows variable names, variable labels 
describing their purpose, and value codes for categorical vari- 
ables, to be stored on the file itself. Since a great number of 
variables are needed in the study, re*forraatting the data will 
prev ::,»-. confusing data items and names. ' 

1. Create SAS File 

Create a separate SK3 file for each data source, (e.g., irs, 
parent interview, student interview), using a SAS program. The 
v^ata items for each file will be contained on one or more records 
(cards), each of which includes a common variable— -either student 
I.D. or institution I.D. Read each record into a separate output 
file and assign variable names to each item. Sort the output 
files by the common variable and merge them to form a single 
file. Because of the number of variables involved, the names 
used should follow the c-/ivention of a three-letter code signi- 
fy! ig the source followed by a two- or three-digit number 
assigned sequentially. For example, variables in the parent 
interview file could be named PAROOl, PAR002, PAR003, etc. 

Translate missing value codes, i.e,, codes for "not appli- 
cable," "no answer," 1 "don't knovf," into a single missing code 
in th - merged file. 
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2. Assign Variable Labels 

Next assign labels to each variable describing the purpose 
of that data item* Labels are used in later analysis procedures 
to produce more readable output- The institution interview file 
may contain many categorical variables with multivalued codes • 
These values should be recorded and labeled in the file as well* 

TASK 6? PRODUCE MARGItlAL TABULATIONS 

After the reformatting of the data is complete / the data 
should be checked again to see that the reformat programs ran 
success fully • If the reformatting programs are done in SAS, it 
is a simple matter to produce marginal tabulations and simple 
statistics as a pairt of these progrcuus* The tabulations may then 
be used to check rhe accuracy and completeness of the data. 
These tabulations provide the analysts with an overall look at 
the range and distribution of the data* In addition, they may 
ser\e as diagnostic tools when problems or questions turn up 
later in Vhe data merge or analysis • 
Procedures ; 

1 • Produce Statistics 

At the end of each reformatting program/ run a procedure to 
determine the frequency of each value in categorical data, (e.g., 
PROC FREQ in SAS)* The distribution of continuous variables, 
including the means, extremes, and quantities, should be found 
using another analysis procedure (e.g*, PROC UNIVARIATE in SAS ) • 
produce cross-tabulations with related categorical variables* 

\ 
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Run at least one of these procedures on every variable. Frequen- 
cies should also be run on ranges of values of continuous vari-* 
ables^ but some care should be exercised as frequencies run on 
continuous variables and univariates run on categorical ^'^ariables 
will produce great quantities of meaningless output- Finally, 
print a number of cases to provide a general checX on the data. 
2* Review 

Review the tabulations r statistics, and print-outs for any 
conspicuous errors. These will most easily be found on frequency 
tables, in cases where the data are obviously not distributed 

V 

properly. A common symptom of problems with either the data'^or 
programs is an inordinate number of missing values for a data 
item. Errors are often detected on print-outs repeated or 
inappropriate values for continuous variables. Errors of this 
type may well have escaped detection in machine edits, or 
possibly were created in the reformatting programs. Even if no 
conspicuous errors are found, these various statistics will be 
useful later as diagnostic tools. 

When any errors are detected their cause(s) should be deter- 
mined and their problera(s) eliminated. This requires checking 
both the reformatting algorithms and the hard copy interview 
data. Finally, the file should be updated, and new marginal 
tabulations, statistics, and prints should be produced. 

TASK 7i MERGE DATA FILES 

In order to analyze error in the Pell Grant program, the col- 
lected data must be manipulated to yield a single file of the 
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best, most raliable data and award calculations • Tfkis file is 
created by a series of merge programs which build the final file 
by retaining values from the various data files • The merge pro- 
grams are ordered in such a way that the most valid data file is 
merged first, the second most valid is merged second , cind so on 
down to the least valid data. In this way datSi items begin with 
missing values and are assigned the most valid available value as 
each new data source is merged. The purposes of the n^erge system 
are: 

• To make final determination of the student's marital 
and dependency status^ and, if dependent/ the marital 
status of the parent 

• To proceed in a step-by-step manner / from most reliable 
to least reliable data files, allowing for intermediate 
checks on the quality and appropriateness of data 

m To provide the "best value" data file from which any 
error computations caxi be made 

Procedures 

Note that the merge programs here have been developed by 
Advanced Technology, and with certain modifications and enhance- 
ments they could be revised for subsequent error studies . 
1. Code Dependency and Marital Status 

The first program in the merge should collect the necessary 
information to code the student's dependency and marital status. 
The status information is necessary to determine what data are 
appropri. te in each case. For example, if a particular student 
is independent, then no parent information should be retained. 
Or, if a dependent student's father has died in the previous 
year, then the merge process must be programmed not to retain his 
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income/ even though it may be reported on some data sources. The 
initial merge program reads the dependency and marital status 
items from all the following files available; parent interviews ^ 
student interviews, and student aid reports. 
2. Prioritize Data 

Any data corroborated by a signed legal document/ such as a 
signed tax form, a certified savings account passbook, or a 
mortgage, should be considered documented data and given priority 
over undocumented data* Such data, however, may or may not be 
available in any given study* If they are available, then the 
data from status and marital questions should be given the fol-- 
lowing priority: larent documented, student documented, parent 
undocumented, student undocumented, and SAR. If documented data 
are not available then the final three categories are the rele- 
vant ones, and their order remains the same* Variables which 
will contain the codes for dependency status (STATUS), students 
marital status (MAR-l), and parents marital status {WAR-2), and 
"flag variables" to code the source of the data, are initialized 
with missing values* As each file is merged, the program fills 
any missing value with the available data. When a case taXes 
value a from a file, it codes the corresponding flag variable. 
After the last file has been merged, STATUS, MAR-1, and, if 
status is dependent, MAR-2 should have values. Any case for 
which these values are still missing must be dropped from the 
sample • 
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3. Merge Files 

The next program in the merge should initialize all vari- 
ables which pertain to tax data (e.g., adjusted gross income or 
medical and dental expenses), as well as the corresponding source 
flag variable, to missing values. The tax data file will now be 
merged to recode these variables with the appropriate values. 
The SAS program will sort out relevant data using the dependency 
and marital status values and a series of IF, THEN DO loops. In 
some cases, data will be irrelevant, such as an independent stu- 
dent's parent tax data, and they will be rejected. Tax data 
should be merged before other data because they are the most 
reliable. Figure 5-6 shows the priority of the data sources for 
the "best values" variables used in the Stage One study and 
indicates the order in which files should be merged. Variable 
names are given along with their descriptions. PARC, STUC, and 
SRAD represent the parent certified, student certified, and 
student record abstract documented files, respectively. PARN, 
STUN, and SRAW represent the parent noncertif ied, student 
noncertified, and SRA weakly documented files. 

Following the IRS file, the parent interview, then student 
interview files should be merged in the same manner. All neces- 
sary "best value" variables have been initialized by this point, 
as well as the means and extremes for some of the continuous 

variables. , 
The next step in the merge is to determine whether student's 
dependency status is different on the best value file from the 
SAR. SAR and Student Record Abstract (SRA) data will not be 
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decreasing priority 



BEST VALUE 

VARIABLE NAME IRS 

CTZN (citizenship) 

MAR-1 {student marital) 

BA (bachelor's degree) 

PAR 80 (Did student live 
with parents in 1980?) 

PAR 81 (in 1981?) 

EX 80 (Did parents claim 
student as exemption in 
1980?) 

EX 81 (in 1981?) 

SP 80 (Did parents give 
student $700 support in 
1980?) 

SP 81 (in 1981?) 

HSHLD (number in 
household) 

INCOL (number in 
college) 

MAR-2 (parent's marital) 

FILED (IRS filed?) 

FILAS (estimated) 

EXMPS (number of 
exemptions) 1 

AGI 80 1 



PARC STUC SRAD PARN STUN SRAV? 

3 



2 
2 



1 
1 



3 
3 



2 
1 



2 
2 



4 
4 



3 
3 

4 
1 

2 
2 



2 
2 
1 
1 

5 
5 



4 
4 

5 
2 

3 
3 



3 
3 
2 
2 

6 
6 



4 
3 



5 
5 



4 
4 



4 
4 



7 
7 



FIGURE 5-6 

DATA MERGE PRIORITY TABLE 
USED T'J STAGE ONE QC STUDY 
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decreasing priority 

BEST VALUE 

VARIABLE NAME IRS PARC STUC SRAD PARN STUN SRAW 

TAX 80 (taxes paid) 1 2 3 4 5 6 7 

ITM 80 (itemized 

deductions) ^ . ^ 3 4 5 6 7 

SS (Social Security 

payments) 1 2 3 4 5 6 

NONTAX (income of head 

of household) 3 12 4 

INC-1 (income of head of 

household) 12 3 4 5 6 

INC- 2 (second income) 1 2 3 4 5 6 

MED 80 (medical/dental 

expenses) 12 3 4 5 6 7 

TUIT (tuition paid for 

high shcool) 12 3 4 5 6 

CASH (cash assets) 12 3 4 5 6 

HOKEV (home value) 1 2 3 4 5 6 

HOMED (home debt) 12 3 4 5 6 

INW (investment value) 1 2 3 4 5 6 

INVD (investment debt) 12 3 4 5 6 

BUSV (business value) 12 3 4 5 6 

8USD (business debt) 12 3 4 5 6 

VAM (monthly VA pay) 12 3 4 

VA-M (number months VA 

pay) 12 3 4 



FIGURE 5-6 (cont'd) 

DATA MERGE PRIORITY TABLE 
USED IN STAGE ONE QC STUDY 
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decreasing priority 



BEST VALUE 

VARIABLE NAME 

SSMO (monthly SS pay) 

SSNMO (number months SS 
pay) 

INC 80 (dependent 
student income) 

ASETS (dependent student 
assets ) 

FRMV (farm value) 

FRMD (farm debt) 



IRS PARC STUC SRAD PARN STUN SRAW 



1 

1 

1 



2 
2 
2 



3 
3 



3 
4 

4 



2 
5 

4 

5 
5 



FIGURE 5-6 (cont'd) 

DATA MERGE PRIORITY TABLE 
USED IN STAGE ONE QC STUDY 
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valid in these cases. The file must be broken into two sets^ one 
containing cases where the status changed , the other containing 
the rest off the cases* The first file, the status changers / 
should then be merged with undocumented data if the documented/ 
undocumented distinction exists. Otherwise ^ the file for these 
cases has all of the "best values" and is ready for the SAI and 
AWARD canputation programs « The second file must be read into 
merge progrstrais to merge data from the SAR file# the SRA file, and 
the undocumented data they are available. 
4« Compute SAI and Award 

The merging of "best value" data is now complete. The re- 
maining tasks to prepare the data for error analysis are to cal-- 
culate the SAI r the correct awards, and the discrepancies between 
these calculated values and the reported values. The prograun to 
calculate SAI will merge the status changer file and will calcu- 
late SAI for each case. The last program will merge the file 
with "best values" and SAI with the institutional interview file, 
from which values for cost of attendance, other expense data, and 
full-time status definitions can be taken. The program should 
then compute the correct award for each student* Finally, the 
program should calculate discrepancies between calculated SAI and 
the SAI reported on the SAR, and between the correct award and 
the award reported on the SRA file* The file is now ready for* 
any error computation and analysis. 

The merge progr2ims require considerable space in the com-- 
puter, a fact which can cause problems for the programmer. The 
sc >pe of the problem depends on the aunount of data collected and 
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coded in the data files* This probl'sm can be minimized by drop- 
ping the incoming data variables from the merged file at the end 
of each program* If the documented/undocumented (or validated/ 
nonvalidated) distinction is operable, then this means that the 
relevant files must be re-merged When the undocumented data are 
to be merged. However/ it may be advantageous to keep certain 
variables When they are merged the first time« For analysis pur- 
poses it is useful to keep the SAR file data as variables sepa- 
rate from the "best values 

TASK 8: CONDUCT QUALITY CONTROL CASE REVIEW 

After the best value file has been created a careful check 
of the data should be made to determine the success of the merge 
programs and to ensure the reliability of the data « It has been 
found that the most effective test is case review, i*e»/ checking 
print outs of cases against the hard copy data* This should be 
done by personnel who thoroughly understand th< data and the sta- 
tistical analyses for Which thye are to be used* The goal of 
case reviewing is, first/ to test the workings of the merge 
programs* In particular: 

# Whether status items were correctly determined 
m Whether •correct values were reported 

# Whether they came from th** correct sources 

Second / the case reviewing tests the overall reliability of the 
data for both random and systematic problems such as: 

# Keypunching errors 

# Coding errors 
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• Interviewer errors 

• Inconsistencies within individual cases 

Cas4 reviewing will usually produce several problems to be cor- 
rected by modifying one or more of the merge programs, as well as 
updates on a case-by-case basis. After changes have been made a 
final case review should be done to be sure no new problems have 
arisen. 
Procedures 
1. Select Sample 

Since the entire population curnot be inspected, select a 
sample of cases to review. There are several ways of selecting 
this sample. One way is to sort the file by award discrepancy 
and take the cases with the largest discrepancies . In this type 
of sample any gross errors, like keypunch errors or merge errors, 
are likely to appear. Another type of sample is a random sample, 
or a random sample stratified by some variable such as disburse- 
ment amount ranges, award discrepancy ranges, AGI ranges, etc 
In these types of 'samples, the systematic errors are more likely 
to appear. The case review should be done with a couple of dif- 
ferent types of samples, if time and budget constraints permit. 
The size of the sample will also depend on these considerations. 

After the sample has been chosen, gather the file and hard 
copy data for those cases- All hard copy data used in the crea- 
tion of the file will be needed (i.e., tax forms, parent and 
student interview data, SAKs, and SRAs). Print all the best 
value variables, the SAR variables used in calculating awards and 



award discrepancies, and the student I.Ds. In addition / include 
the source flag variables that form the best value file* 
2. Review and Update 

Compare the hard copy data with the file data. Be careful 
to find the best value for each variable in the documentation wO 
check against the file. In reviewing the cases, one must be well 
versed in all the possible mitigating circumstances which cause 
some otherwise valid- looking values to be passed over. 

When an error is found, determine whether it is a systematic 
error or a keypunch/editing/codir^ error. Systematic errors are 
those which , occur every time the same conditions are present in a 
case# For example , if all dependent students with a recently 
widowed parent have two different parent incomes on the file, 
then it is assumed that a systematic error is occurring. 

If e- » w's are found, correct the file through an update of 
the merge programs, or an ad hoc merge program which selects the 
affected cases and corrects them. Keypunch/editing/coding errors 
can be individually corrected on the file itself, although other 
errors of this type will remain* If a great number of these 
types of errors are found* then closer investigations are needed 
to see if they are, in fact, due to some systematic error. 

After all the necessary program changes and file updates 
have been made the previous error cases should be checked to see 
that the problems have been solved. A new sample is then drawn 
and the cases reviewed to see that new problems have not arisen. 



CHAPTER 6 

GUIDELINES FOR CONDUCTING DATA ANALYSIS 

PRELIMINARY STATISTICAL ANALYSIS 

Grouping Unit Recordg for Analysis 

Once error values have been assigned to each sample case, 

rules for including cases in suitimary statistical runs should be 

specif ied« These rules should address the following issues; 

m Inclusion or deletion of cases (unit records) where no 
disbursements are recorded 

• Inclusion or deletion of cases where data collection 
is incomplete 

• Inclusion or deletion of cases where verification data 
are incomplete 

Specification of inclusion rules involves the considera- 
tions detailed below. 

Zero Disbursement Cases 

There are several reasons why there may be no recorded Pell 
award disbursement for a given case. For examp-le, (1) the 
student may have been considered ineligible for a grant by the 
institution, and therefore no grant was awarded; (2) the student 
may have withdrawn frcsn school before picking up his or her 
award; (3) the student may have decided not to pick up the award 
for personal reasons; and (4) there may have been an error in 
institutional disbursement records. Different explanations for 
zero disbursement have different implications for analysis. In 
some cases, zero disbursement reflects correct institutional 
behavior, and in others it may reflect either an error in 
bookkeeping or administrative procedures . 
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Analyst* should specify which option is to be used in choos- 

jif ' Which cases ^ if any, should be deleted in error analyses: 

Option 1j Delete all unit records with zero disburse- 
ment data or missing disbursement data . 

Discussion: Cases with no recorded disbursement do not 
represent recipient cases; if the study is restricted 
to error in the recipient population, cases with no 
award disbursement should be deleted from analyses . 

Option 2i Delete cases selectively after case review. 

Discussion: If institutional data permit, case-by-case 
review can determine whether disbursements were with- 
held by institations because students were deemed 
ineligible. These cases could be retained in file, 
because error calculationj take eligibility criteria 
into consideration. Cases Where students fail to pick 
up awards for various personal reasons would be deleted 
from the file. This option provides a more accurate 
neasure of the correctness of institutional procedures 
among SAI-eligible applicants. There is also reason to 
retain these cases if measurements of student error 
only are being made, since disbursement errors or miss- 
ing disbursciment data are irrelevant to measures of 
student error. 

Incomplete File Data 

Cases where data collection is not complete may not be 
appropriate for analysis. Specifications must be set out for 
cases in Which different types of data collection are not present 
in the merged data files. 

All possibilities for missing data types should be speci- 
fied. This can be done using a Guttman scale-type system, where 
the most essential data collections, probably students' official 
aid indexes, are rated most important, and each other data 
source, e.g., IRS data, interview data, student record data, is 
ranked in priority order. If the most important data are missing 
for a case, the case should be deleted from the study 



regardless of \Aat other ciata have been collected. Decision 
rules for a study with five data sources — SARs, tax forms, parent 
and student interviews, and institutional data — might then be 
specified as follows: 

• Cases missing SARs (or disbursement SAIs) are deleted. 

• Cases where institutional data files are missing are 
deleted * 

• For dependent students, cases missing both parent 
interview and parents* IRS forms are deleted. 

• For independent- filing students, cases missing both 
student intervie%ra and student tax forms are deleted, 
unless pareni. interviews are present and indicate that 
students are dependent. 

Incon^lete or Missing Verification Data 

The most serious inclusion/exclusion decision to be made is 

the decision regarding measurement of error in cases Where full 

verification data have not been obtained for all items. If the 

confirmatory approach to verification is tal<en (as discussed in 

Chapter 2), no (zero) error can be calculated in cases where no 

verification data exist. Cases Where no documentation has been 

coded can be identifed hy the flags created in the data files. 

Analysts therefore choose between several t^tions; 

Option 1; Select all otherwise complete cases, allow- 
ing zero error to be counted in cases with no documen- 
tation (e.g., no verified values). 

Discussions This selection option will provide iJie 
roost accurate measure of error caught by the methodol- 
ogy used in the study* 

Option 2: Select only cases Where flag variables 
indicate that selected items (e.g., AGI, dependency 
status) have been verified. 
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Discussion: This option provides an estimate of error 
among the population for whom documentaticm has been 
provided, e.g*, all recipients for %4hom a 1040 has 
been collected. Such an estimate will be a biased 
one, because the population for whom tax forms have 
been obtained will differ from those for whom a tax 
form has not been obtained. Thus, estimates drawn 
using this selection procedure cannot be applied to 
the entire population of recipients vrithout analysis 
of the sampling bias involved and appropriate adjust- 
ment of the estimates. 

Option 3» Perform initial analyses as in Option 1, 
then assign the mean error measure to cases where no 
verification/documentation has been collected. This 
can be done by singly adding an adjustment factor (the 
proportion of unverified cases times mean error) to 
the total measured error for the population after 
analytical progreuas have been run on all available 
cases. 

Discussion: This option provides an estimate of total 
population error Which is based on the hypothesis that 
error in undocumented (or unverifiiKi) cases is equal 
to error in documented cases. Analyses raay indicate 
that this hypothesis is incorrect. In that event, the 
adjustment factor must be changed. 

Assigning Error in Categorically Ineligible Cases 

The formulae for measuring overall payment error in the Pell 
program (Chapter 2) differentiate student and institutional 
error. Categorically ineligible recipients create special mea- 
surement problems because entire disbursements are counted as 
institutional error. In some instances, analysts may want to 
measure payment error without assigning total disbursements to 
ineligible recipients, or specific types of ineligible recipi- 
ents, as institutional error. 

Institutional data include specific items relating to all 
categorical eligibility. These may be used to create a single 
eligibility item. The following coding system assigns 10 unique 
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eligibility scores related to all the categorical eligibility 
crit^^ria: 

0 « eligible by all categorical criteria 

1 = no valid statement of educational purpose 

2 « no financial aid transcript 

3 « student holds B-A- degree' 

4 » not a citizen or eligible U.S. resident 

5 » not enrolled at least half time 

6 » not in a course of at least six months duration 

7 « in default on a Federal loan 

8 ^ not making satis factoid academic progress 

9 « not in a Pell-qualified program 

Using these codes analysts can then select the cases in which 
all disbursements should be assigned as institutional error* 

This system makes it possible to assign error differently in 
different calculations, e.g./ to not measure Pell payment error 
in cases where SEPs or FATs are missing, or to totally disregard 
institutional categorical error and to measure only student error 
and institutional error related to disbursements calculation, and 
accounting procedures* (See Chapter 2 for a discussion of 
different types of Institutional error*) 
Generating Descriptive Statistics 

Basic measures of overall payment error may be presented in 
a variety of ways . The most important descriptive statistics to 
be produced are the univariate statistics which show the inci-- 
dance and general form of overall payment error. 
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The basic univariate measures and frequcKicies should be com- 
puted for all payment error variables. The most iroportant 
measures can be presented as frequencies: 

Cases Where: 

Absolute payment error Verified award ^ 

Net payment error Disbursed award 

Overpt-yraent error Disbursement > verified 

award 

m 

Underpayment error Disbursement < verified 

award 

Standard statistical packages normally produce frequency tables 
showing the frequency (case count), cumulative frequency, sample 
percentage, and cumulative percentage represented by each fre- 
quency category. Analysts may want to report counts or percen- 
tages (e.g., percent of the sample with error), or frequencies of 
the size of error within specified ranges. Figure 6-1 illus- 
trates a shell for illustrating simple frequencies of overall 
error in the Pell program. Fig 6-2 shows a shell for report- 
ing frequencies grouped into ranges of award error in dollars in 
1980-81. 

Standard statistical software packages (SAS, SPSS, etc.) 
also generate a series of univariate statistics and graphic rep- 
resentations of univariate distributions. Analysts will want to 
review the appropriate univariate statistics — means, medians, 
standard deviations, skewness, etc.— in order to detemine the 
nature of the distribution of error. Particularly useful summary 
statistics can be generated quite easily with standard statisti- 
cal packages (e.g., SUMMARY and UNIVARIATE in SAS, CONDESCRIPTIVK 
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ERROR TYPE 



ESTIMATED % OF RECIPIENTS 
WITH THIS ERRORS 



1 . Student Error 

2. Bachelor's Degree or 
Citizenship Error 

3. SEP or FAT Error 

4. Program Eligibility Error 

%* Cost of Attendance Error 

\ 

\ 

6. ^ Enrollment Status Error 

7. Calculation Error 
Sum of All Errors 



^Individual recipients may have more than one type of error. 
Therefore, individual error rates do not add up to the total. 



FIGURE 6-1 

EXAMPLE TABLE SHELL 
FOR REPORTING SIMPLE FREQUENCIES 
OF OVERALL PAYMENT ERROR 
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PERCENTAGE OF CASES 



STUDENT 6 

ALL STUDENT INSTITUTION ERROR STUDENT ERROR 

& INSTITUTION NO*^ INCLUDED NOT INCLUDING 

AWARD ERROR ERROR AEP/FAT ERROR AEP/PAT ERROR 



- $551 


and : 


less 


- 9251 


to - 




- $151 


to - 


$250 


- $51 


to - 


$150 


- $3 


to - 


$50 


$2 


to - 


$2 


$3 


to 


$50 


$51 


to 


$150 


$151 


to 


$250 


$251 


to 


$550 


More 


than 


$550 



FIGURE 6-2 

EXAMPLE TABLE SHELL 
FOR REPORTING FREQUENCIES GROUPED 
INTO RANGES OF AWARD ERROR 
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in SPSS). Statistical packages also produce various graphics, 
such as bar charts (histograms) or pie charts which depict fre- 
quency disbtributions in readily understandable ways. Error fre- 
quencies may be run for particular subpopulations, e.g., depen- 
dent students or nonvalidated students. Mean error statistics 
for subpopulations are also useful. Figure 6-3 shows a sample 
bar chart (the height of the bars is purely illustrative) of the 
absolute mean dollar error for institutions rather than for 
students . The ranges represented by the bars are determined by 
the analysts ; tLs height of the bars represents the relative 
frequency of observations within each range. 

BIVARIATE ANALYSIS 

The first step in investigating the correlates or causes of 
payment error is to determine whether specific student, institu- 
tion, or program characteristics are associated with the inci- 
dence or size of payment errors. Bivariate analyses, i.e., 
two-way tables and/or statistics measuring the strength of asso- 
ciation between two variables, can be extremely useful in both 
answering questions about specific relationships and in explora- 
tory data analysis. Contingency tables (crosstabs) can be used 
to evaluate the validity of hypotheses regarding the causes of 
error in the Pell program. Because the factors contributing to 
student error may be totally different from those related to 
institutional error, separate bivariate analyses should be run 
for each type of error as well as for overall payment error 
categories. 
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FIGWC 6-3 

EXAMPLE BAR CHART 
FOR REPORT INS ABSOLUTE HEAR 
DOLLAR ERROR FOR iNSTITUTIONS 
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A number of factor* associated with payment arror can b« 
drawn from pr«viou« studios of arror in ths ?mXl systsm. D«p«n- 
dency status, incoras, family size, type of school attended, etc., 
may be associated with the incidence of overpayment error among 
students. Institutions I error rates may differ among public, 
private, or proprietary institutions, or among different ^ypes of 
institutions (universities, colleges, junior colleges, or less 
than two-year programs ) . Student or institutional error rates 
may also be associated with different institutional procedures, 
e.g., institutional validation, use of automated procedures, or 
institutional QC checks. The size, professional composition, or 
training of financial aid and staffs may also be factors. Pre- 
vious Pell studies provide a large number of hypotheses regarding 
the factors associated with error in the Pell program. Analysts 
may wish to explore other possible factors as well. 

Contingency tables can also be used as a fairly crude means 
of exploratory data analysis. Analysts may decide to run a rela- 
tively large number of tables simply to see if any interesting 
and unexpected relationships show up. Generating large numbers 
of samples using standard statistical packages (assuming offx- 
cient programming) is not particularly expensive and may prove 
quite useful, especially in early stages of analysis. The review 
of a series of tables may alert analysts to unusual relationships 
in the data Which should be investigated more thoroughly. 

Organizing Bivariate Analysis 

Analysts should identify the tables they would like to pro- 
duce in a systematic way in order to reduce computer costs. 
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Initial bivariat* runo can be specified by listing student, 
institution, and program characteristics Which analystis have 
reason to believe may be aa<icciated with payment error. For each 
of these independent variables , analysts can then select which 
error measurement — total error, student error, institution 
error — and What categories of error (ranges, means, etc.) they 
want to look at. An example of a matrix of cross tabulations is 
shcvn in Figure 6-4. The list of characteristics or factors 
possibly associated with error is merely illustrative and is in 
no way designed to bo an exhaustive listing of useful bivariate 
&ialyaes. Each "X" represents a table Which would produce mean- 
ingful data for analysts seeking to under starvi the causes of 
error in the Pell program. Analysts should sptocify the particu- 
lar ranges and/or statistics they would like to see for each 
table in this type of matrix in order to maximize proo'-araming 
eff icxency. 

Measures of Association for Two-Way Contingency Tables 
Standard statistical packages allow analysts to generate a 
wide range of test statistics along with contingency tables 
(cross-tabulations). Those statistics are designed to allow 
analysts to choose the appropriate test 3tatistic(s) when deter- 
mining the existence or strength of the relationship between the 
variables being evaluated. Since statistics, including chisquare 
(< ) ectablish only whether the variables are statistically depen- 
dent or independent . Others , such as I-amda (^ ) measure the 
strength of the association of nominal (categorical) variables. 
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Ranges (Size) Ranges (Size) 
Total Error Student Error 



Ranges (Size) 
Institution Error 



Student Characteristics 



Oependenciy Status X X 

Income (ranges) X X 

Filed Tax Forms (yes/no) X X 

Tyf>e of Asiets Reported X X 
Demographics 

Age XX 

Year In school X X 

Family size X X 

Region X X 



Institution Characteristics 

Type (4-year, 2-year, < 2 year) 

Control (public, private, proprietory) 

Size (enrollment) 

Number of Pell Recipients 

Size of Financial Aid Staff 

OSFA Training 

Procedures 

Institutional validation 

Collect documentation 

Automated systems 

QC system In place 

Program Characteristics 

Validation (random, PEC) 
MDE Processor 

Recent Program Audits, Reviews 



XX X 

XX X 

XX X 

XX X 

XX X 

X X X 

X X 
X X 

XX X 

XX X 



XX X 
X X 

XX X 



FIGURE 6-4 



EXAMPLES OF TO-WAY CONTINGENCY TABLES FOR 
ANALYSIS OF OVERAU PAYMENT ERROR 
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Thi« atatistic, for •xaraple, would be useful in interpreting data 
relating dependency statue to the existence of overpayments, 
underpayments, or correct award disbursements. Other statistics, 
such as gamma, take the order, or direction, of table categories 
into account, so that scores reflect both the strength of the 
relationship and the way that one variable (e.g., AGI group) is 
associated with another (e.g., sixe of payment error category). 
Analysts should be careful to choose appropriate test statistics 
when evaluating contingency tables. 

MULTIVARIATE ANALYSIS 

Multivariate analysis needa to be differentiated from the 
bivariate techniques discussed above. Bivariate techniques test 
for the existence of differences in error levels among groups of 
students, where grouping would be baaed on a single charactar- 
i3tic of the student, his or her family, or the institution 
attended regardless of the differences across the groups. Thus 
the two variables involved in a bivariate analysis are (1) error 
level and (2) the single case characteristic. 

Multivariate techniques are designed to address more compli- 
cated (but conditional or contingent) questions of relationships 
between error and characteristics of students, families, -ind 
institutions. Rather than asking if a relationship exists be- 
tween error and a single characteristic, multivariate analysis 
asks Whether a relationship exists between error and a single 
characteristic while allowing relationships between error and 
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other variables to simultaneously exist. Put another way, multi- 
variate models assume or allow that error is affected or deter- 
mined by a multitude of characteristics. As discussed in the 
following, it is necessary to make assumptions about the form of 
this multiple determination in order to effectively utilize 
multivariate analysis. 

In a program as complicated as Pell Grants, and in one 
involving complex human behavior, it is unlikely that the simple 
relationships assumed in bivariate analysis exist. However, care 
must be exercised when using multivariate techniques since, while 
they do not assume simple relationships, they do require assump- 
tions about the form, structure, and nature of the complex multi- 
ple interactions among error and characteristics. 

There are three purposes which can be served by multivariate 
analysis : 

• Testing of a priori hypotheses 

• Exploratory data analysis 

• Error-prone modeling 

The first two represent the two methodological approaches 
to empirical research. Error-prone modeling represents fiui appli- 
cation of multivariate modeling to the development of decision 
rules. 

Hypothesis testiug is a methodological approach where hypo- 
theses and theories are subjected to real world data In order to 
confirm or reject these hypotheses. Exploratory analysis, on the 
other hand, uses the real world data to develop the hypotheses 
and theories* 
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Hypothaaig Tqgtinq — Hypothesis testing involves four steps t 

• Enumeration or specification of policy-related 
hypotheses 

e Selection of appropriate test statistics 

• Specification of the model which comes from the theory 

• Estimation and testing 

Developing the list or set of hypotheses to be tested 
depends on numerous sources* One source would he previous Pell 
Grant error studies. A second source would be relevant financial 
aid literature* Policy and program experts can also supply pol- 
icy-related questions which can be expanded by suggestions from 
budget and planning personnel. Generally, a hypothesis would 
involve testing for the existence of a relationship between level 
of error or probability of error and a student, family, or 
institutional characteristic. 

Each hypothesis test can result in four outcomes; 

e Reject a true hypothesis 

e Accept a false hypothesis 

• Reject a false hypothesis 

• Accept a true hypothesis 

The last two outcc»nes represent correct decisions while the 
first two represent erroneous decisions — type I and type II 
errors, respectively. , It would be desirable to select or create 
decision rules ani te^t statistics which minimized the chances of 
these two types of ^rors. However, with a given sample sire 
reducing the chance of making a type I error increases the chance 
of making a type II error, and vice versa. In order to determine 
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the optimal trade-off between the chances of making the two types 
of errors requires knowledge of the cost and/or consequences of 
each type of error* Thus^ one must know the consequences of 
accepting the existence of a relationship when, in truth, one 
does not exist and of rejecting the existence of a relationship 
when, in truths such a relationship does exist « 

Model specification plays a crucial role in multivariate 
analysis because any relationship beina tested is assumed to 
exist in the presence of many other relationships ♦ The model 
specifies the form of such relationships, and the tests focus on 
the existence of that particular type of relationship • For 
example, one might assume that characteristics are nearly reduced 
to error measures, perform a test, and conclude a relationship 
does not exist. However, if the relationship was multiplicative, 
experimental, or logarithmic, such a conclusion would be false 
since the truth is that a linear relationship does not exist. In 
the extreme, rejecting a hypothesis about the existence of a 
relationship requires rejectii^ all possible forms the relation- 
ship might take on* Therefore, any conclusions drawn are condi- 
tional on choice of functional form, specification of the error 
structure, and choice of the characteristics jointly considered 



to affect error levels 

Once decisions concerning the hypothesis, the decision rule 
trade-offs, and nK>del specification have been settled, the selec- 
tion of statistical instructing techniques and methods remains • 
Figure 6-5 presents characteristics of various multivariate 
techniques. In Pell Grant error studies the dependent variable 
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Type of Dependent Type of Independent 
Technique Variable (y) Variables (x) Conronts 



Multiple 

Regression 

(Ordlrdiry 

least 

squares) 



A. Continuous 



DIChOtOflKHiS (0,1), 
continuous, or 
mixed. Equivalent 
to AflOVA If all 
dlchotooous and to 
ANCOVA If Mixed. 



Probit 



A. Dichotoflious 

B, Special version 
for N'Chotoiiious 
(ordinal) avail- 
able 



01 chotomous • contin- 
uous, or nixed. If 
all d1 chotofsous , 
contingency tables 
are cheaper and 
sinpler. 



Logit 



Same as probit 



Sane as probit 



u 

Discriminant OlchotOfiious Supposed to be all 

Analysis continuous and dis- 

tributed fliiltlvarl- 
ate norwal , but this 
is very comnonly 
violated. 



Basic technique 
with variations: 
steptrfde, weight- 
ed, restricted, 
tine-series, 
equation sys- 
tems. 

Interpretation a 
bit messier than 
logit. 



Easier to Inter- 
pret than probit. 
More robust to 
assumption viola- 
tions than dls- 
crlnant analysis. 

Allows user to 
Incorporate a 
priori esti rates 
of event proba- 
bilities to make 
estimates more 
efficient. Esti- 
mation programs 
more refined to 
give displays, 
measures of good- 
ness of fit, etc. 



CHARACTERISTICS OF 



FIGURE 6-5 

VARIOUS MULTIVARIATE TECHNIQUES 



Type of Dependent Type of Independent 
Technique Variable (y) Variables (x) Cowiients 



Automatic 

Interaction 

Detection 

(AID) 



Contlmious. A 
version, called 
THAID. exists for 
dlchotoRTHis case. 



ANOVA (One- 
way) 



Continuous 



AfiOVA 
(K-way) 



Continuous 



Dichotonous or N- 
chotoiKHiS. (Can 
recode continuous 
variables to achieve 
this.) 



A single ordinal or 
categorical vari- 
able. (Representing 
that variable as a 
set of dlchotoMOus 
variables In wltl- 
ple regression is 
equivalent. ) 

K ordinal or cate- 
gorical variables. 
(Representing each 
variable as a set of 
dichotoMOUs vari- 
ables in Multiple 
regresslcHi Is equiv- 
alent. If interac- 
tion terns are 
Included.) 



Should not use If 
N is less than 
3000. Technique 
is "stepwise 
ANOVA. " Best 
used in "fishing" 
for a set of 
independent vari- 
ables and inter- 
action tens to 
be Included in a 
regression. 



FIGURE 6-5 

CHARACTERISTICS OF VARIOUS MULTIVARIATE TECHNIQUES (Cont'd) 
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would b« level of error (continuous) or the existence of error 
(dichototmous) . Independent variables would include students 
family » and institution characteristics. 

Exploratory Analysis — This type of multivariate analysis 
differs from hypothesis testing in that here the relationships 
which exist in the data are determined. Li hypothesis testing 
one asXs Whether certain pre-established relationships exist in 
the data. Therefore, in exploratory analysis theories are built 
from data. Whereas in hypothesis testing theories are tested 
against data. 

Two basic methods are recommended for exploratory data 
analysis: 

• Automatic Interaction Detector 
e Stepwise multiple regression 

Both these methods are embedded in existing software pack- 
ages which define search algorithms. As such, these analyses can 
be replicated by other data analysis. 

The AID program involves a sequential search for binominal 
splits leading to maximum exploratory power or predictive models. 
Stepwise regression can use a number of search algorithms, such 
as: 

• Forward inclusion 

• Backward exclusion 

• Maximum predictive ability 

• Forward exclusion witn backward exclusion 

• Minimum predictive ability for Inclusion 
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It is also po8sibl«i to usa any multivariate statistical 
program and develop customized or researcher-defined search 
methods . 

Error-Prone Modeling (EPM) — EPt4 is a form of exploratory 
analysis; however / it differs from the methods just discussed in 
that its purpose is not to uncover relationships among variables 
but rather to split a sample into groups Where the observations 
in a group have as similar error levels as possible while error 
levels across groups are as dissimilar as possible. 

Once these gzroups are defined they can be ranked by error 
levels as presented in Figure 6-6 which yfa.8 taXen from the Stage 
One Quality Control report. Group 35 had the highest average 
error ($381) representing 4.1 percent of total error and 1.0 
percent of the students. Group 16 had the lowest net overpayment 
(21), representing about 4 percent of total error, but nearly 20 
percent of the students. 

Definitions of these groups are presented in Figure 6-7. 

These data can then be used to develop a Lorenz-type curve 
relating cumulative error and cumulative cases. This method of 
data presentation can be used to develop validation rules and 
levels of effort. For example, if the policymaker wanted to 
remove 40 percent of new overaward, this could be accomplished by 
validating about 15 percent of the cases. These 15 percent would 
be groups 35, 27, 33, 29, 37, 31, 34, and 32, the first through 
eighth ranked groups. An example of a Lorenz-type curve is shown 
in Figure 6-8. 
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GROUP CUMUUTIVE Ntf«ER CUMUUTIVE NUMBER 

NUMBER NET ERROR NET ERROR % OF CASES OF CASES % 
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FIGURE 6-6 

EXAMPLE TABLE 
FOR REPORTING ERR0R-PR0P£ GROUPS 
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APPENDIX A 



GLOSSARY 



AD t Actual Disbursement The ainount of Pell Grant money that a 
student receives n "iven ^ear. Should not be confused 
with expected di& -'i ment or scheduled award. 

ADS I Alternate Disbursement System. One of two methods by which 
Pell Grants are paid. Because of size or other factors 
Which affect a school's ability to process award payments, 
a student is paid his or her Pell Grant by the Department 
of Education instead of by the institution the student 
attends . 

AGIi Adjusted Gross Income. An item on the Pell application. 
One of the primary factors considered in determining a 
student's eligibility for a Pell Grant. The application 
asks for the AGI as reported on IRS Form 1040 or 1040A. 
For most applicants, AGI is the total income earned from 
work, plus interest incon», dividends, and other taxable 
income . 

BA Error I Bachelor's Degree Error. Students with bachelor's 

degrees are ineligible for Pell Grants. BA error is con- 
sidered one of the ccnnponents of institutional error since 
certifying whether or not a student has received a BA is 
the institution's responsibility. 

Business or Farm Value/Pebt t Two items on the Pell application. 
The first refers to the market worth of the applicant's 
(for independents) or parent's (for dependents) business 
and farm at the time of application. The second refers to 
how much is owed at the time of application. 

Central Processor ! The firm, under contract with the Department 
of Education, which receives and processes all student 
application data and %#hich produces and distributes SARs. 

Citizenship Error t Students who are not U.S. citizens, U.S. 
nationals, or permanent residents of the U.S. or its 
territories are ineligible for Pell Grants. Citizenship 
errc>r is considered one of the ccmiponents of institutional 
error since institutions are responsible for certifying an 
applicant's citizenship prior to the disbursement of a 
grant . 

cost or COA « Cost of Attendance. Cost is the total of a 

student ' s actual tuition and fees, room and board expenses, 
plus a $400 allowance for books and supplies. It is one of 
three factors used to calculate an expected disbursement — 
a student's enrollment status and SAI being the other two. 
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Cost error occurs when the calculated cost fi^re used to 
calculate a student's award does not equal that student's 
actual cost* tt is considered a component of institutional 
error. 

Course Length Error * To be eligible for a Pell Grant a student 
must be enrolled in a program of study at least six months 
in length, course length error is considered a component 
of institutional error. 

Dependent Racipilnt i h student receiving a Pell Grant as well as 
significant financial sui^rt from his or her parent. For 
the 1981-82 Pell Grant award year, dependents were con- 
sidered those students who met at least one of the follow- 
ing criteria > 

1. Lived with a parent for more than six weeXs during 1980 
or 1981 

2. Was claimed as an exemption on a parent's 1980 or 1981 
tax form 

3. Received more than $1,000 of support from a parent in 
1980 or 1981 

Eligible Program of Study Erro r t To be eligible for a Pell Grant 
a student n»st be enrolled in a program which leads to a 
bachelor's, associate, undergraduate pr'^fessional, or cer- 
tificate degree. Eligible program of study error- is con- 
sidered a component of institutional error. 

Enroll I Enrollment Status. All Pell Grant recipients are con- 
sidered enrolled on either a full-time, three-quarter time, 
or half-time basis. Enrollment status is one of three 
factors used to calculate a Pell Grant expected disburse- 
ment. 

Expected Disbursement ! The aiiKTunt of Pell Grant money expected 
to be received by a student during an award period, based 
on the student's SAX, enrollment status, and cost of atten- 
dance. Expected disbursements do not necessarily corres- 
pond with actual disbursements. 

Family Size t An item on the Pell application. For dependents, 
includes all people in the parent's household (including 
the parents) who will receive at least half of their sup- 
port from the parents during the Pell Grant award year. 
For independents, includes all people in the student and 
spouse's household (including the student and spouse) who 
will receive at least half of their support from the 
student and spouse. 
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FAT Error t Financial Aid Transcript Error. Before a transfer 

student may receive a Pell Grant, the institution must have 
received and evaluated a certified financial aid transcript, 
from that student's prior educational institution. FAT 
error is considered a component of institutional error 
since it is the institution's responsibility to withhold a 
grant disbursement if ah PAT has not been received. 

Father '.s/A^gUcant' 8 Inconw t An item on the Pall, application- - 
For dependents, refers to the amount of income the appli- 
cant's father earned from work? for independents, refers to 
the amount of incone the applicant earned from work. 

Grant or Loan Default Error i A student is not eligible to 

receive a Pell Grant if he or she owes a repaymeiit on any 
Title IV grant or is in default on any Title IV loan 
received at that institution. Grant or loan default error 
is considered a con^nent of institutional error since it 
is the institution's responsibility to certify that the 
student does not owe a repayment on a grant or is in 
default. 

Half-time Enrollnwnt Status Error i To be eligible for a Pell 

Grant a student must be enrolled at least half-time. Since 
it is the institution's responsibility to certify enroll- 
ment status prior to a disbursement, half-time enrollment 
status error is considered a component of institutional 
error. 

Home Value /Debt I Two items on the Pell application. The first 

refers to the market worth of the applicant's (for incepen- 
dents) or parent's (for dependents) home at the time of 
application. The second refers to the anK>unt owed on the 
home at the tin» of application. 

Independent Recipien t! A student receiving a Pell Grant who is 
not dependent on his or l^er family for financial support. 
For the 1981-82 Pell Grant award year, independents were 
considered those students vfho had nots 

1. Lived with a parent for six weeks in 1980 or 1931 

2. Been claimed as an exemption on a pai w'nt's 1980 or 1981 
tax form 

3. Received more than $1,000 of support from a parent in 
1980 or 1981 

Medical and Dental Expenses » An item on the Pell application. 

Refers to the amount paid for medical and dental expense,; 
not covered by insurance. 
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Mothor * 8 /Spou se ' 8 Incon» t An item on the Pell application. For 
dependients, refers to the amount of income the applicant's 
mother earned from work? for independents, refers to the 
amount of income the applicant's spouse earned from worK. 

Nontaxable Inces t The sum of three items on the Pell applicat* 
tions Social Security Benefits, Aid, to Families with 
Dependent Children (AFDC), and Other Nontaxable Income. 
Other Nontaxable Income includes child support, earnings 
from work not reported on a tax return, unen^loyment com- 
pensation, disability income, and interest on tax free 
bonds • 

OSFAt Office of Student Financial Assistance. OSFA is respon- 
sible for administering the five major student assistance 
programs s Pell Grant, Guaranteed Student Loan (GSL), 
National Direct Student Loan (NDSL), College Work-Study 
(CW-S), and Supplemental Educational Opportunity Grant 
(SEOG). OSFA has a functional organisation structure, with 
management responsibilities assigned according to function 
rather than program. Thore are six divisions! 

DPPD* Division of Policy and Procjram Development 

DPO t Division of Program Operations 

DCPR t Division of Certification and Program Review 

DSDD t Division of Systems Design and Itevelopment 

DTD* Division of Training and Development 

DOA » Division of Quality Assurance 

Other A.ssets /Debts t Two items on the Pell application. The 

first refers to the market worth of the applicant's (for 
undependents ) or parent's (for dependents) real estate arid 
investments and the time of application. The second refers 
to how much is owed at the time of application. Invest- 
ments include trust funds, stocks, bonds, and other 
securities! « 

PIMSj Program Information Monitoring System. An automated 

system which monitors the allocation and obligation of Pell 
Grant funds to institutions, and through institutions, the 
disbursement of funds to students. 

Progress Report * A document completed by all institutions par- 
ticipating in the Pell program which is submitted a^. least 
quarterly to the Department of Education. On the Progress 
Report, the institution details current program expendi- 
tures. The report enables the Depc^rtment of Education to 
determine if the institution's annual authorization should 
be raised or lowered. 

RDS i Regular Disbursement System. Method by which most students 
are paid Pell Grants. Institutions on the Regular Dis- 
bursement System receive funds from the government/ with 
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thm /amount rdceived based on the projected num^r of 
•tiidents attending the institution* Students receive Pell 
Gr^pt payments directly from the institution* 

1/ 

SAIx Stiji^dent Aid Index. Number given to applicant, based on 

ap|>licant*8 financial strength as indicated by factors such 
aSjAGX, Federal taxes paid, family size, and assets. The 
SaIe is^ combined with the applicant's cost of attend'^nce and 
eiJrollment status to determine his or her grant level, 

SAR ; Stmdent Aid Report • A report returned to the applicant 

frcxn the Central Processor after the application has been 
submitted and processed. The SAR provides the applicant 
with an SAX* The student must sulmit the SAR to the 
institution he or she plans to attend before a Pell Grant 
will be awarded* 

Satisfactory Academic Progres.s Error ; To be eligible for a Pell 
Grant a student must maintain satisfactory progress in his 
or her course of study* Each institution sets its own 
standard of satisfactory academic progress, and it is the 
institution's responsibility to certify that progress is 
being maintained before it disburses a grant. Satisfactory 
Academic Progress error is considered a component of insti- 
tutional error. 

Scheduled Award ? The amount that a full-time student enrolled 
for a full academic year is entitled to receive, 

SEP Error ; Statement of Educational Purpose Error- To be eli- 
gible for a Pell Grant, a student must file a statement 
with his or her institution stating that all funds received 
through Title IV programs will be used solely for educa- 
tional or educationally related purposes* SEP error is 
considered a component of institutional error. 

Student ' s /Spouse ' s Exjggcted Income t Refers to the sum of four 
items on the Pell application « Includes all income the 
applicant and spouse expect to receive during the summer 
prior to the academic year and during the acadmic year 
itself. 

Student *s /spouse's Net Assets i An item on the Pell application 
completed by dependent students only* Refers to all the 
applicant's and spouse's assets (hc^iie, investments, 
business, farm^ and savings) minus what is owed on those 
assets. 

Student ' s /Spouse ' s Net Incoiw % An item on the Pell application 
complet^ed by dependent students only. Refers to all the 
applicant's and spouse's incomie (taxable and nontaxable) 
minus the U,S. income taxes that were paid. 
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Taxes Paid i An item on the Pell application. The application 
asks for the amount of Federal taxes paid as reported on 
the IRS Form 1040 or 1040A. 

Tuition » An item on the Pell application. Refers to the amount 
of money the applicant's family paid for elementary, junior 
high, and high school tuition. 

Validation < The process by which a portion of Pell Grant appli- 
cants are selected and required to present to their finan- 
cial aid officer certain documents such as IRS Form 1040 or 
1040 and W-2 statements, which confirm the accuracy of the 
information on the Pell application. 

« 
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APP£t7DXX B 
MEASUREMENT TOLERANCES 

Measurement tolerances in the 1979-80 and 1980-81 studies 
of error in the Pell Grant program were drawn very tightly. 
Tolerances for dollar error in application data vere two dollars 
in both stuaies. Discrepancies between application values and 
verified values of greater than t%^ dollars are reported as item 
errors in the study reports. This narrow tolerance bancl was 
adopted for several reasons: 

1. Much of the available verification is reported, or 
recorded on documents, in exact dollar values. 

2. Computing with exact dollar values is neither more dif- 
ficult nor more expensive than using rounded figures . 

3. The SAI computation formula provides a complicated 
rovinding scheme; rounding application items before 
entering them into the formula might well produce dif- 
ferent SAX scores. 

Since exact application item figures are needed for SAI computa- 
tions, and since collection of exact figures presents no serious 
data collection, editing, or computer processing problems (in 
fact, rounding by interviews and coders creates more opportunity 
for error), it seemed reasonable to make application item error 
tolerances as tight as the data permitted? the t%#o dollar toler- 
ance was designed to allow for roundin<:} problems related to 
reporting figures down to the penny. 

Reporting of overall payment error figures in the two Pell 
studies was designed to take these narrow tolerances into consi- 
deration. Overall payment error and item error figures were 
reported not only in total, based on the $2 tolerance, but by 
ranges, e.g., -within $2," "$2-$50,'' "^Sl-^lOO." 

B-1 
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APPENDIX C 

ALGEBRAIC SPECIFICATION OF ERROR MEASURES 

In order to more clearly define error measures # algebraic 
specifications have been presented in this appendix. Table C-1 
presents the notation used in the following sections. 

In addition to recorded values of expected disbursement it 
is also possible to calculate an expected disbursement using 
value of enrollment status* student aid index # cost of atten- 
dance # and categorical criteria. The following notation is used 
to represent calculated expected disbursement based on the value 
of SAI at time of data source n, cost of attendance recorded at 
time m, enrollment status from source and categorical criteria 
obslerved at time 1. 

EDCSAl(n), COA(m), ENR(k), CAT(l)3 
Using the above notation we specify post-reconciliation 
ovetall error as follows: 

Disb(*) - EDCSAK*), C0A{*), ENR(*)]. 
For pre-reconciliation ov^all error the above formula would be 
roqdified by using DISB(3) in place of Disb{*). 

This total error can be decomposed into student error ^md 
institution error as indicated below. 
\ Total Error > 

Disb(*) - edCsaK*), coa(*), enr{*)3 

1 student Error < 

\ 

edCsai(4), C0A(*/, enr(*)3 
-edCsai(*), coa(*), enr{*)3 
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Institution Error : 

Disb(*) - EDCSAI(4), C0A(*), KNR(*)3 

Enrollraent Status Error i 

EDCSM(4), C0A(4), ENR(4)3 
-EDCSAI(4), C0A(4), ENR(*)3 

Calculation Error » 

Di8b(*) - EDCSAI(4), C0A{4), ENR(4)] 
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Item 

Enrollment Status 
Cost of Attendance 
Student Aid Index 
01 sbursement 
Expected Disbursement 
Scheduled Award 
Categorical Criteria 



Initial 
SAR 



Fall 
SAR 



ENR(l) 
COA(l) 
SAI(o) SAI(l) 

E0(1) 
SA(I) 
CAT{1) 



Student i 
Pa rent 
Interv1e«fs 



SAI(*) 



dsbursemenc 
SAR 

ENR(3) 

C0A(3) 

SAI(3) 

E0(3) 
SA(3) 
CAT{3) 



Institution 
Record 
Abstract 

EMR(*) 

COA(*) 



DISB(*3) 



Reconclllatlof 
Roster 

Eltm(4) 

C0A(4) 

SAH4) 

DISB(4) 



CAT{*) 



FIGURE C-1 
TERMS USED IN ERROR DEFINITIONS 
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AN ESTIMATE OF RESOURCE AND SCHEDULE REQUIREMENTS FOR 
CONDUCTING AN ASSESSMENT OF OVERALL PAYMENT ERROR IN 

THE PELL GRANT PROGRAM 

This appendix presents estimates of the resource and tinne requirements for 
completing a Pell Grant payment error study. Erased on a review of several options for 
collecting the data to routinely measure overall payment error in the Pell Grant 
program 1, it was recomm«ided that 05FA continue to use the three-faceted approach 
that was employed in the 197&-79 and 1980-Sl Quality Control studies. This approach 
includes: (I) student record reviews at a sample of postsecoi^^y institutions; (2) field 
interviews and audits of a sample of aid recipients and their parents; and (3) collection 
of documentation directly from various governm^t agencies which verifies certain 
student application information. 
Resource Requirements 

Figure D-l {N'ovides estimates of resource requirements for five alternative 
samples of recipients and institutions. These estimates assume an average of 15 
recipients per institution. Resource requirements have been classified as direct labor 



^The following four optlcxn were identified in Chapter h 

• Option 1 . Student record data would be collected by site visits to 
institutions; students and parents would be interviewed in person; and 
docunoentation would be collected by maU from the IRS and other agencies, 
(The recommended approach.) 

• Option 2. Would have the same features as Option 1 except that in-person 
interviews with students and parents would be replaced by telephone 
interviews. 

• Option 3 . Would be a scaled-down version of Options 1 and 2'-no student 
and parent interviews would be conducted. 

• Option ^ , Would have no field work; all institutional and student/parent 
data would be collected by mail. 
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SELECTED SAMPLES 



1 



RESOURCE REQUIREMENTS! 



2»500 RecipienU 
167 bistitutions 



3,900 Recipients 
260 Institutkms 



9,500 Recipients 
367 bistitutiofis 



7,100 Recipients 
♦73 Institutioro 



3,700 Recipients 
MO institutions 



LABOR (in hours) 

Senior Staff 
Mid-Level Staff 2 
Junior Staff' 

Total Hours 



3,100 
S,OQO 
3^,»0O 



3,100 
9,500 
»8,000 

60,600 



3,100 
11,300 
63,600 

78,000 



3,100 
13,000 
79,200 

95,300 



3,100 
14,700 
9»,800 

112,600 



i 



OTHER DIRECT COSTS (in $)♦ 
Travel 

Telephofie Time 
Reproduction 
Pos tage/Suppt ies 
Data Entry 
Computer Time 
Other 

Total Other Direct Costs 



$123,900 
14,806 
24,200 
11,300 
18,300 
36,000 
9,800 

$238,300 



$165,200 
22,700 
34,000 
17,300 
27,300 
40,500 
10,300 

$317,300 



$212,500 
32,300 
45,200 
24,200 
40,300 
45,600 
10,800 

$410,900 



FIGURE D-l 

SUMMARY OF ESTIMATED RESOURCE REqUlREMENTS 
FOR FIVE SELECTED SAMPLES^ 
SELECTION OF SAMPLE PRIOR TO SITE VISITS 



$259,600 
41,600 
56,400 
31,100 
52,000 
50,700 
11,300 

$502,700 



$304,000 
49,800 
67,600 
38,000 
62,600 
55,300 
11,900 

$589,200 



* Figures in this table are rounded. For a detailed analysis af labor costs refer to Appendix D-1. For an analysis of other direct costs, 
refer to Appendix D-2, . 

^This labor category includes hours for institutional data collectors. 



^ 'This labor category includes hours for student/parent interviewers. 
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costs, expressed in hours, and other direct costs (ODC's), expressed in 1982 dollars. 
Three labor categories are identified: senior, mid-level, and junior. Senior staff 
include Ph.D.'s and other personnel with significant experience in survey research, 
computer science, statistics, sociology, education, or economics. Mid-levei staff 
include those at the M.A. or B.A. level with one to six years of relevant experience. 
Institutional field workers are included in this category, Junior staff include research 
assistants, clerks, secretaries, and student and parent interviewers. 

The following assumptions were made in order to estimate resource requirements 
for the five selected samples* 

• The recipient sample would be drawn by securing a list of each institution's 
recipients prior to the site visits. 

• An institution visit would consiit of 15 record reviews and one <^5 minute 
interview. 

• The quantity of information collected at each institution and from each 
student and parent would be apfN-oximately the same as the quantity 
collected during the 1980-81 study. 

• The institution sample would be cli»tered geographically. 

• Two hundred field staff would interview the students and parents. 

• The student/parent interviewers would be regionally based; the institu- 
tional field workers would not be regionally based. 

• The analysis requirements would approximate those of the 1980-81 study. 

• A replication of the approach used in the 1980-81 study would be conducted 
with lower development costs and fewer inefficiencies. Instruments, 
training manuals, procedures numuals, and computer programs were 
developed for the Stage One study. With modifications, these materials 
could be reused for future error studies. In addition, the Technical 
Specifications document would serve as a comprehesive procedures manual. 

The policy maker and budget planner must keep these assumptions in mind when 

interpreting the figures in Figure D-l. Changes in the assumptions can change costs 

and can result in changes in resource requirements. For instance, decreasing the 

amount of Tnformation required froi.. the student and parent interviews would 

substantially decrease junior staff labor costs and travel expenses. 
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Figure D-l reveals that total direct labor hours increase substantially with an 
increase in the sample size— a 200 percent increase in the recipient sample size 
roughly translates into a 100 percent increase in labor hours. A closer examination of 
the figures, however, indicates that an increase in junior staff (student/parent 
interviewers and clerks) hours accounts for most of the change. The hours required for 
higher salaried mid-level and senior staff are not nearly as sensitive to changes in 
sample size. Therefore, a substantial increase in labor hours due to an increase in 
sample size <kies not iiecessarily mean a substantial increase in the total cost of 
project staff salaries. 

Likewise, Figure D^l reveals that certain ODC items are more sensitive to 
sampie changes than others* Travel expenses— which incliKie airfare, rental car costs, 
and per diem— vary substantially with changes in sample size while costs associated 
with compKiter time do not. 

The estimates in Figure D-i assume that the recipient sample would be drawn 
prior to the institutional site visits. Figure D-2 provides estimates if the recipient 
sampie is Jrawn during the site visits by the institutional field workers. A comparison 
of Figure 0-1 and D-2 reveals that the on-site sample selection approach would be 
more costly to the government than the pre-visit selection approach. Although senior 
and junior staff requirements are somewhat less with the on-site af^oach, mid-level 
staff needs are significantly greater due to the increased amoiait of field work. 
Overall ODCs are .slightly greater with the on-site approach, also due to the increased 
field time. 

Of particular interest is the level of effort required of institutional field 
workers, given the possibility of diverting— or hiring additional— DC PR program 
reviewers to perform this task. It is estimated that each institutional visit would 
require 13 hours of field worker time if the pre-visit sample selection approach was 
used and 20 hours if the sample was chosen on-site, tJsing these benchmarks, the 
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BEST COPV 



SELECTED SAMPLES 



12 I « $ 



2«500 Recipients 3,900 Recipients 5,500 Recipients 7,100 Recipients S,700 Recipients 
RESOURCE REQUIREMENTSi 167 Institutions 260 institutions 367 institutions %73 institutions 580 bistitutions 



LABOR (tn hours) 



Senior Staff 
Mid-Level Staff 2 
Junior Staff3 

Total Hours 



3,000 
9,100 
34,200 

W,300 



3,000 
11,200 
»7,800 

62,000 



3,000 
13,800 
63,»00 

80,200 



3,000 
16,200 
79,000 

98,200 



3,000 
18,600 
9»,600 

116,200 



f 

1/1 



OTHER DIRECT COSTS (in $)♦ 

✓ 

Travel 

Telephone Time 
Reproduction 
Postage/Supplies 
Data Entry 
Computer Time 
Otlter 

Total Other Direct Costs 



$135,200 
13,500 
24,200 
11,200 
18,300 
36,000 
9,800 

$248,200 



$183,900 
20,900 
34,000 
17,100 
27,300 
40,500 
10,300 

$33'»,000 



$238,900 
29,300 
45,200 
23,800 
40,300 
45,600 
1J,800 

$433,900 



$293,700 
37,800 

%,m 

30,700 
52,000 
50,700 
11,300 

$532,600 



$348,700 
46,200 
67,600 
37,600 
62,600 
55,300 
11,900 

$629,900 



i^URE D-2 

SUMMARY OF ESTIMATED RESOURCE REQUIREMENTS 
FOR FIVE SELECTED SAMPLES t 
ON-SITE SAMPLE SELECTION 



•figures in this table are rounded. For a detailed analysis of. tabor costs refer to Appendix D-I. For an analysis of other direct costs, 
refer to Appendix D-2, ^ ^ 

^This labor category includes hours for Institutional data collectors. f^,,^ 

'l^^J^Xm labor category includes hours (or student/parent interviewers. 
tiVlL^ 



following are estimates of required field worker hours for each of the selected samples 
and for both sample selection approaches. 

Selection of Sample Prior to Site Visits 

Sample 1-*2,171 hoin^ (271 person days) 

Sample 2^3,380 hours {^23 person days) 

Sample 3—^,771 hours (3% person days) 

Sample <^6,l(f9 hoi^ (769 person days) 

Sample !^7f3¥} hours (943 person days) 

Selection of Sampie On^te 

Sample 1—3,3^ hours (41S person days) 

Sample 2—3,200 hours (650 person days) 

Sample 3— 7,3W) hours (918 person days) 

Sample ^9,460 rwurs (1,183 person days) 

Sample 5—11,600 hours Hf^50 person days) 

If OSFA were to undertake the field data collection using OSFA personnel, then 
these figures could be used to estimate the labor requirements for data collection 
efforts of each sample size. This would require either hiring temporary staf. or 
releasing current regional staff to undertake the data collection. In either case 
training would be required for the data collectors. 

A more entailed analysis of labor requirements is found in Appendix D-1. 
Appendix D-2 contains a detailed breakdown of ODCs. These two appendices contain 
formulas which the budget planner can use to estimate costs for studies with sample 
sizes other than the five listed in figures D-l and D-2. In Appendix 0-3 guidelines 
for estimating costs for collecting student and parent interview information by 
telephone are presented. 
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Schedule Requirements 

The following series of tables show recommended schedules for conducting the 
institutional data collection portion of the error study. Figures D-3-1A to D-3-1D 
present recomnicnded timeframes for conducting a study using the fjre-visit sample 
selection approach for 10, 13, 20, 20 and 25 data collectors. Figures E)-3-2A to D-3- 
2D show schedules for 10, 15, 20, and 25 data collectors when an on-site selection 
approach is used. Each figure presents estimated schedules for the five selected 
samples. The estimated schedules have been divided into three stiKly phases. The pre 
data collection phase includes the sample selection, instrument development, and 
recruitment and training of the field staff. The data collection phase includes all the 
field work and the supervision of the field work. The preparation and analysis of the 
data are included in the pKjst data collection phase. 

The estimates in the figures are subject to the same assumptions that were listed 
for figures D-1 and D-2. Most notably, the time periods listed for the data collection 
phase assume the completi<Ki of three institutions per week if the recipient sample is 
selected prior to the site visits and the completion of two institutions per week if the 
sample is selected on-site. 



ERIC 



234 



D-7 



SELECTED SAMPLES 





1 


2 


3 


4 


5 


STUDY PHASE 


2,900 Recipients 
167 Institutions 


3,900 Recipients 
260 Institutions 


5,500 Recipients 
367 btstitutkms 


7,100 Recipients 
♦73 Institutions 


8,700 Recipients 
580 institutkxis 


Pre Data Collection 


10 Weeks 


10 Weeks 


11 Weeks 


11 Weeks 


12 Weeks 


Data Collection* 


6 Weeks 


9 Weeks 


12 Weeks 


16 Weeks 


19 Weeks 


Post Data Collection 


8 Weeks 


8 Weeks 


8 Weeks 


8 Weeks 


8 Weeks 


TOTAL 


2t Weeks 


27 Weeks 


31 Weeks 


35 Weeks 


39 Weeks 



FIGURE D-5-1A 

RECOMMENDED TIMEFRAMES FOR COMPLETING DATA COLLECTION AT INSTITUTIONS 

BY SELECTED SAMPLES : 
10 DATA COLLECTORS WITH SAMPLE SELECTED BEFORE SITE VISIT 

* Assiiifoe cdcfi data Collector completes an average o( 3 institutions per week. 

236 



SELECTED SAMPLES 





1 


2 


1 




5 


STUDY PHASE 


2.X)0 Recipients 
167 institutions 


3.900 Recioients 
260 Institutions 


RecJuients 
367 institutions 


7.100 Recioimts 
*73 institutions 


580 Institutkms 


Pre Data Collection 


10 Weeks 


10 Weeks 


U Weeks 


tl Weeks 


12 Weeks 


Data Collection I 


4 Weeks 


6 Weeks 


8 Weeks 


1 1 Weeks 


13 Weeks 


Post Data Collection 


8 Weeks 


8 A^eeks 


8 Weeks 


8 Weeks 


8 Weeks 


TOTAL 


22 Weeks 


2<f Weeks 


27 Weeks 


30 Weeks 


33 Weeks 



o 
I 

VP 



FiGURE D-3-IB 

RECOMMENDED TIMEFRAMES FOR COMPLETING DATA COLLECTION AT INSTITUTIONS 

, BY SELECTED SAMPLES 
15 DATA COLLECTORS WjTH SAMPLE SELECTED BEFORE SITE VISIT 

• Assume each data collector completes an average of 3 institutions per week. 
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SELECTED SAMPLES 



STUDY PHASE 



1 



2,500 Recipients 
167 Institutions 



3,900 Recipients 
260 Institutions 



5,500 Recipients 
367 Institutions 



7,100 Recipients 
473 institutions 



S,7O0 Recipients 
5S0 Institutions 



Pre Data Collection 
Data CoUectionJ 
Post Data Collection 

TOTAL 



10 Weeks 
3 Weeks 
8 Weeks 

21 Weeks 



10 Weeks 
5 Weeks 
% Weeks 

23 Weeks 



11 Weeks 
6 Weeks 
8 Weeks 

25 Weeks 



11 Weeks 
8 Weeks 
8 Weeks 

27 Weeks 



12 Weeks 
10 Weeks 
8 Weeks 

30 Weeks 



I 



FIGURE D-3-1C 

RECOMMENDED TIMEFRAMES FOR COMPLETING DATA COLLECTION AT INSTITUTIONS 

BY SELECTED SAMPLES 
20 DATA COLLECTORS WITH SAMPLE SELECTED BEFORE SITE VISIT 
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* Assume each data collector completes an average of 3 institutions per week^ 
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SELECTED SAMPLES 





1 


2 


3 




> 


STUDY PHASE 


2,300 Recipients 
167 institutions 


3,900 Recipients 
260 bntitutions 


5»900 Recipients 
367 Institutions 


7,100 Recipients 
473 Institutions 


8,700 Recipients 
580 Institutions 


Pre Data Cot lection 


10 Weeks 


10 Weeks 


11 Weeks 


1 1 Weeks 


12 Weeks 


Data Collection! 


2.5 Weeks 


It Weeks 


5 Weeks 


6 Weeks 


8 Weeks 


Post Data Collection 


8 Weeks 


8 Weeks 


8 Weeks 


8 Weeks 


8 Weeks 


TOTAL 


20,5 Weeks 


22 Weeks 


lit Weeks 


25 Weeks 


28 Weeks 



FIGURE D^>-ID 

RECOMMENDED TIMEFRAMES FOR COMPLETING DATA COLLECTION AT INSTITUTIONS 

BY SELECTED SAMPLESi 
25 DATA COLLECTORS WITH SAMPLE SELECTED BEFORE SITE VISIT 

Assume each data collector completes an average of 3 institutions per week. 
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SELECTED SAMPLES 





1 


2 


3 


# 


9 


STUDY PHASE 


2,500 Recipients 
167 kistitutions 


3,900 Recipients 
260 institutions 


5,900 Recipients 
367 institutions 


7,100 Recipients 
«73 kistitutkMis 


8,700 Recipients 
5S0 institutions 


Pre Data Collection 


7 Weeks 


7 Weeks 


7 Weeks 


7 Weeks 


7 Weeks 


Data Collection 1 


8 Weeks 


13 Weeks 


IS Weeks 


24 Weeks 


29 Weeks 


Post Data Collection 


8 Weeks 


8 Weeks 


8 Weeks 


8 Wcdcs 


8 Weeks 


TOTAL 


23 Weeks 


28 Weeks 


33 Weeks 


39 Weeks 


Weeks 



o 
I 



FIGURE i>-3-2A 

RECOMMENDED TIMEFRAMES FOR COMPLETING DATA COLLECTION AT INSnTUTIONS 

BY SELECTED SAMPLES* 
to DATA COLLECTORS WITH SAMPLE SELECTED ON-SITE 

'Assume each data collector compldtes an average of 2 institutt«>ns per week. 
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SELECTED SAMPLES 





1 


2 


3 


4 


5 


STUDY PHASE 


Ay^UV Kecipi^ts 

167 Institutions 


JyTUII Kectfraenis 
260 Institutions 


7y>uu Kecipienis 
367 Institutkms 


/ yiuu Kccipicnis 
473 institutions 


9 7/111 Da^S««]a«%#« 

9y/iiu Keci|Nenis 
$80 liotitutions 


Pre Data Collection 


7 Weeks 


7 Weeks 


7 Weeks 


7 Week 


7 Weeks 


Data Coltectionl 


6 Weeks 


9 Weeks 


12 Weeks 


16 Wect'S 


19 Weeks 


Post Data Collection y 


8 Weeks 


8 Weeks 


8 Weeks 


8 Weeks 


8 Weeks 


TOTAL 


21 Weeks 


Ht Weeks 


27 Weeks 


11 Weeks 


Weeks 



I 

I-" 



FIGURE D-3-2B 

RECOMMENDED TIMEFRAMES FOR COMPLETING DATA COLLECTI ON AT INSTITUTIONS 

BY SELECTED SAMPLES^ 
15 DATA COLLECTORS WITH SAMPLE SELECTED ON SITE 

1 Assume cach\iata collector completes an average of 2 institutions per week. 
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SELECTED SAMPLES 





1 


2 


3 




5 


STUnV PHASE 


167 Institutions 


7(7wu Kcciptenis 
260 Institutions 


juu Keciptenis 
367 Institutions 


/fiuu KfscifMents 
473 institutions 


8,700 Keclpients 
580 Institutions 


Pre Data Collection 


7 Weeks 


7 Weeks 


7 Weeks 


7 We^ks 


7 We^U 


Data Collection I 


H Weeks 


7 Weeks 


9 Weeks 


12 Weeks 


H Weeks 


Post Data Collection 


8 Weeks 


8 Weeks 


8 Weeks 


8 Weeks 


8 Weeks 


TOTAL 


19 Weeks 


22 Weeks 


21* Weeks 


27 Weeks 


30 Weeks 



u 
I 



PiGURE i>-3-2C 

RECOMMENDED TIMEFftA»flES FOR COMPLETING DATA COLLECTION AT iNSTITUTiONS 

BY SELECTED SAMPLESi 
20 DATA COLLECTORS WITH SAMPLE SELECTED ON SITE 



^Assume each data collector completes an average of 2 institutions per week* 



247 



248 



SELECTED SAMPLES 





1 


2 


3 




5 


STUDY PHASE 


2,500 Recipients 
167 institutions 


3,900 Recipients 
260 Institutions 


5,500 Recipients 
367 bistitutions 


7,100 Recipients 
473 Institutions 


8,700 Recipients 
980 Institutions 


Pre Data Collection 


7 Weeks 


7 Weeks 


7 Weeks 


7 Weeks 


7 Weeks 


Data Collection^ 


3.5 Weeks 


5 Weeks 


7 Weeks 


10 Weeks 


12 Weeks 


Post Data Collection 


8 Weeks 


8 Weeks 


8 Weeks 


8 Weeks 


8 Weeks 


TOTAL 


18.5 Weeks 


20 Weeks 


22 Weeks 


25 Weeks 


27 Weeks 



f 

H 



nCURE 0-y-2D 

RECOMMENDED TIMEFRAMES FOR COMPLETING DATA COLLECTION AT INSTITUTIONS 

BY SELECTED SAMPLES? 
25 DATA COLLECTORS WITH SAMPLE SELECTED ON SITE 

'Assume each data collector completes an average of 2 institutions per week. 
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APPENDIX D-i 



LABOR REQUIREMENTS BY 
TASK AND LABOR CATEGORY 
(In Person Hours) 



TASK 



LABOR CATEGORY 



VARIABLE COST TASKSi 



Senior 
Staff 



Mid 
Level 



3uniar 
Staff 



Totai 



Schedule Institution Site Visits 

Collect Data at Institutions2,3 

Interview Students/ Parents* 

Supervise Institution Data Collectors , '^/^fs-/ 

Institution 

Supervise Student/ Parent Interviewers 
Code/ Edit Institution Forms 

Code/ Edit Student/ Parent Forms 

Reiease/Receive/File 
Secondary Forms^ 

Code/Edit Secondary Forms 



.2 hrs./ 
Institution 

13<20) hrs./ 
Institution 



.3 hrs./ 
Institution 

.1 hrs./ 
Interview 



.2 hrs./ 
Institution 



hrs./ 
Interview 



.1 hrs./ 
Interview 

5 hrs./ 
Institution 

.7 hrs./ 
Interview 

.25 hrs./ 
Instrument 

.33 hrs./ 
Instrument 



FIXED COST TASKS 



Select Sample^ 


60S 


264 


320 


1,1^^2 




(500) 


(150) 


(100) 


(750) 


Develop Instruments 


552 


6^*0 


80 


1,272 


Recruit Institution Data Collectors 


50 


10 


5 


65 


Recruit Student/ Parent Interviewers 


UO 


30 


15 


205 


Train/Debrief Institution Data 










Collectors^ 


200 


880 


160 


1,2^0 


Train Student/ Parent Interviewers^ 


260 


520 


8,4^f0 


9,220 


Analyze/Report Findings 


1,200 


3,000 


1,000 


5,200 



TOTAL FIXED HOURS^ 



3,030 
(2,922) 



5,34/> 
(5,230) 



10,020 
(9,800) 



18,394 
(17,952) 



ERLC 
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APPENDIX D^l 



LABOR REQUIREMENTS BY 
TASK AND LABOR CATEGORY 
(continued) 

1 Marginal costs expressed in this tabic assume a recipient sample between 2,000 and 
9,000 and an institution sample between 100 and 600. 

2 Assumes the following: 

• An institution site visit inclu<tes 1 5 record reviews and a ^5-minute interview. 

• The same quantity of information is collected from each student record as was 
collected in the Stage One stiKiy. 

• The institution sample is clustered geographically. 

• The interviewers are not regionally based. 

^Use numbei- out of parenthesis if sample is selected from lists received from 
institutions prior to site visits. Use number in parenthesis if sample is selected on-site. 

^Assumes the following: 

• The same quantity of information is collected during each interview as was 
collected in the Stage One stu^y. 

• Two hundred interviewers conduct the interviews regardless of the sample 
size. 

• The interviewers are regionally based. 

The unit cost here is the completed student aid parent interview; the number of 
completed interviews is calculated as follows: 

Recipient sample x 2 x .S7 (the percent of completes in the Stage One QC study) 

'The number of secondary data collection instruments is calculated from the recipient 
size as follows: 

Recipient sample x 1.8 

^Includes 5 days of training and one day of defriefing for 10 interviewers. Total labor 
hours will increase slightly with a larger interviewing staff. Additional labor costs 
include planning, preparation of materials, and training by mid-level and senior staff. 

^Includes 5 days x 200 interviewers for training. /»dditional labor costs include planning, 
preparation of materials, and training by mid-level and senior staff. 
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APPENDIX D-2 

OTHER DIRECT COSTS 
BY TASK I 



VARIABLE COST TASKS 

Select Sample 

Telephone Tinie2,3 
Postkge3 

Develop lostrunnents 

Reproduction* 

Schedule Inrtitution Site Visits 

Telephone Time 
Postage 

Collect Data at Institutions 

Air Travel 
Car Rental3 
Subsistence' 
Postage 

Interview Studwits/Parents 

Travel^ 
Postage 

Supervise Institution Data Collectors 
Telephone Time 

St^)ervise Stu nit/Parent Interviewers 
Telephone Time 



.5(.l) hrs. per institution 3 $20 per hr. 
<K0) letters per institution @ 20c P««* letter 



25 pgs. per instrument @ 7c per pg. 



.5 hrs. per institution Q $20 per hr. 

3 letters per institution @ 20C per letter 



$S0 per institution 

1. 6(2,5) days per institution 3 $20 per day 
1.6(2.5) days per institution 3 $60 per day 
I package per institution (3 $5 per package 



$9 per interview 

I package per interview @ $2 per package 



.75 hrs. per institution @ $20 per hr. 



.1 hr. per interview (3 $20 per hr. 



ERIC 



^Ali estimates of <joUars and cents costs are based on 1982 prices. 

2The $20 per hour rate assumes a WATS or similar system, 

'Use number out of parenthesis if sample is selected from lists 
received from institutions prior to site visits. !Jse number in parenthesis if 
sample is selected on-site. 

^Calculate the nu.fil>er of instruments to be printed as follows; 
Recipient sample size x * 

'includes air travel, car nsntalf and subsistence. 

D-18 
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APPENDIX D-.2 

OTHER DIRECT COSTS 
BYTiyaC 
(continued) 



VARIABLE COST TASKS (continuc»d) 

Code/Edit Institution Forms 

Data Entry6 
Computer Time 

Code/Edit Student/Parent Forms 

Data Entry 
Computer Time 

Release/Receive/FUe Secondary Forms 

Postage^ 
RetrievaJ* 

Code/Edit Secondary Forms 

Data Entry 
Computer Time 



158 cards per institution @ 25C per card 
.03 CPU hrs. per institution @ $550 per hr. 



9 cards per interview @ 25C per card 
.002 CPU hrs. per interview @ $550 per hr. 



S0< per request form 
$1 per request 



I card p-r instrument (§ 25C per card 

.0001 CPU hrs. per instrument @ $550 per hr. 



FIXED COST TASKS 



Select Sample 

Reproduction 

Computer Time 

Develop Instruments 

Local Travel 



25 copies of sampling plan (3 50 pgs. each 

(a 7c/page = $ 88 

One hour @ $550/hr. = $ 550 



12 trips for field testing @ W miles each 

@ 22.5</mUe = $ iQg 



*1 5 record abstracts @ 10 cards each plus 1 interview questionnaire g 8 cards s 158 
cards per institution. 

^Calculate the number of request forms as follows: 
Recipient sample x .5 

^Calculate the number of request fees as follows; 
Recipient sample x ,33 
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APPENDIX D-2 

OTHER DIRECT COSTS 
BY TASK 
(continued) 



FIXED COST TASKS (continued) 

Recruit butitution Data CoUectors 

Advertising 
Air Travel 
Car Rental 
Subsistence 
Telephone Time 

Recruit Student/Parent Interviewers 

Advertising 
Telephone Time 

Train/Debrief Institution Data CoUectors 

Air Travel 

Lodging 

Reproduction 

Supplies 

Train Student/ Parent Interviewers 

Reproduction 

Supplies 

Stgaervise Institution l^ta Collectors 

Air Travel 
Car Rental 
Subsistence 

Supervise Student/Parent Interviewers 

Air Travel 
Car Rental 
Subsistence 



6 advertisements @ $l,00O/ad. 

3 trips @ $370/trip 

3 days <§ $35/day 

3 days ^ $60/day 

20 calU (§ .5 hr$./call (3 $20/hr. 



3 advertisements ^ $l,0OO/ad, 
10 calls 3 .5 hrs./call (§ $20/hr. 



10 interviewers Q 2 trips each 
@ $373/trip 

10 interviewers 3 6 nights each 
@ $*5/night 

50 copies of training manuals (9 

70 pages each @ 7c/page 

10 interviewers ^ $1 5/interviewer 



250 copies of training manuals 

(3 250 pgs. each Q 7c/page 

200 interviewers @ $2/interviewer 



20 trips @ $370/trip 
*0 days $35/day 
'^O days (§ $60/day 



3 $ 6,000 
c $ 1,110 

=: $ 105 

= $ 180 
^ $ 200 



= i 3,000 
= $ 100 



$ 7,^^00 
$ 2,700 



$ 
$ 



2<^5 
150 



48 trips (a $370/xrip 
96 days @ $3Vday 
96 days 3 $60/day 



$ *,375 
$ W)0 



$ 7, WO 
$ l,«^00 
$ 2,400 



$17,760 
$ 3,360 
$ 5,760 
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APPENDIX I>>2 



OTHER DIRECT COSTS 
BY TASK 

FIXED COST TASKS (continued) 



Analyza/Report Data 



Computer Time 
Reproduction 


' 50 CPU hrs. @ $550/hr. 
15 copies of Draft Analysis Plan 
@ 50 pgs. each @ 7c/page 


= $27,500 
= $ 53 




15 a^ies of Final Analysis Plan 
@ 50 pgs. each @ 7C/page 


= $ 53 




15 copies of Draft Final Report 
0 *00 pgs. each 0 7c/page 


= $ 420 




50 copies of Final Report 
@ ^00 pgs, each ^ 7<i/page 


= $ 1,400 


Locatl Travel 


40 trips (3 49 -Tiiles each 
Q 22.5<i/mile 


= $ 360 


TOTAL FIXED OTHER DIRECT COSTS 




$94,577 



256 
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GUIDELINES FOR ESTIMATING 
RESOURCE REQUIREMENTS FOR 
TELEPHONE INTERVIEWING OF 
STUDENTS AND PARENTS 



VARIABLE 



FIXED 



U Interviewing: IJ5 hrs./phooe interview compared 
to * hrs./in-person interview 

2, Coding/Editing: .5 hrs./j^ione interview compared 
to J hrs*/in-person interview 



!• Training; 3,280 totai hoars to train interviewing 
staff (assumes *0 trainees) cofnpared to 9,220 
hours to train field interviewers (assames 200 
trainees) 



OTHER DIRECT COSTS 



VARIABLE 



Interviewing; Telephone time replaces travel as 
the major cost item associated with interviewing: 
L5 hrs./interview @ $20/hr. 

2. Supervision of Interviewers: No travel and long 
distance telephone time associated with 
supervision of telephone interviewers 



FIXED 



I. Survey tNianagement System: $10,000 

2* Tratining; Since there are fewer phone interviewers 
(assume <fO) than field interviewers (assume 200) to 
train, reproduction and supply costs are lower: 

Reproduction for training $ 1,573 

Supplies $ 80 



r^, ^ ♦^f 



